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ABSTRACT 


This  Report  summarizes  recent  activities  in  the  Department 
of  Defense  and  in  the  US  Navy,  Army,  and  Air  Force  to  establish 
Service  use  of  Interactive  Electronic  Technical  Manuals  (lETMs) 
as  replacements  for  paper  Technical  Manuals  for  logistic 
support  of  military  equipment. 

The  IETM  concept  is  described,  and  an  overview  is  provided 
of  five  IETM  acquisition  Specifications  and  Military  Handbooks 
developed  by  the  Tri-Service  Interactive  Electronic  Technical 
Manual  Working  Group  established  in  1989  by  the  Defense  Quality 
and  Standardization  Office. 

One  of  these  five  draft  documents,  MIL-M- I ETMQA ,  Quality 
Assurance  (QA)  Program  Requirements  for  Interactive  Electronic 
Technical  Mctnuals  (lETMs)  and  Associated  Technical  Information, 
1  Jun  1990,  is  described  and  presented.  (Four  other  companion 
Reports  have  been  prepared  to  introduce  and  describe  the  four 
related  IETM  acquisition  Specifications  and  Handbooks.) 

This  Report  (1)  summarizes  the  Quality  Assurance  Program 
philosophy  upon  which  MIL-M-IETMQA  is  based;  (2)  introduces 
the  concept  of  the  Quality  Assurance  Program  Plan  (QAPP) ; 

(3)  summarizes  the  difference  in  Quality  Assurance  procedures 
required  for  lETMs  and  those  used  in  preparing  paper  TMs; 

(4)  describes  and  explains  the  particular  QA  procedures 
required  for  IETM  production  by  MIL-M-IETMQA  (particularly  in 
the  area  of  Validation  of  the  IETM  System,  the  IETM  Source  Data 
Base,  and  the  View  Package  production  process) ;  and 

(5)  summarizes  proposed  DOD  acceptance  tests  (in  an  Appendix) 
for  information  and  guidance. 

A  copy  of  MIL-M-IETMQA  is  included  in  this  Report  as  an 
Appendix. 
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1.0  INTRODUCTION 


1.1  BACKGROUND 

During  the  1980s,  it  became  increasingly  apparent  that  the 
striking  increases  in  the  complexity  and  sophistication  of  the 
weapon  systems  of  all  three  Services  were  causing  a  serious  lag 
in  the  production,  distribution,  and  management  of  the 
Technical  Information  required  to  maintain,  operate,  and 
support  these  systems.  Of  particular  concern  were  increasing 
weight  and  space  requirements  resulting  from  the  increasing 
bulk  of  the  required  paper  Technical  Manuals. 


At  the  same  time,  a  number  of  significant  technological 
improvements  were  being  made  in  the  field  of  information 
handling,  particularly  the  advent  of  small,  inexpensive,  fast 
computers.  Such  innovations  offered  the  potential  of  almost 
complete  replacement  of  paper-based  Technical  Information 
through  the  use  of  light,  easily  stored,  highly  capable 
electronically  processible  media,  which  at  the  same  time  were 
capable  of  more  effective  interactive  display  to  the  end  user. 


Research,  Development,  Test,  and  Evaluation  efforts  of  the 
three  Services  during  this  past  decade  have  conclusively 
demonstrated,  both  through  field  tests  and  through  in-house 
analyses  and  experimentation,  the  feasibility  and  intrinsic 
value  of  providing  integrated  Technical  Information  in 
paperless  form  in  such  a  way  that  it  can  be  displayed  to  end 
users  by  means  of  an  interactive  Electronic  Display  System. 
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For  example,  the  Navy  Technical  Information  Presentation 
System  (NTIPS)  Program  at  David  Taylor  Researcn  Center,  the 
Navy's  Lead  Laboratory  for  TI  automation,  demonstrated  under 
operational  conditions  the  improvements  achievable  in 
maintenance-technician  performance  [Refs  (1)  and  (2)]  through 
the  use  of  electronically  displayed  TI.  Similar  results  have 
been  achieved  by  the  Air  Force  under  its  Computer-based 
Maintenance  Aiding  Information  System  (CMAS)  and  its  Integrated 
Maintenance  Information  System  (IMIS)  programs 
[Refs  (3)  and  (4)].  The  Army  has  automated  Training 
Information  under  its  Electronic  Information  Delivery  System 
(EIDS)  ,  and  has  assessed  the  capability  of  using  field  p>  .-table 
maintenance  aids  under  the  Militarized  Electronic  Information 
Delivery  System  (MEIDS)  program. 

In  addition,  a  number  of  pilot  prototype  developments  and 
tests  involving  land,  sea,  and  air  vehicles  and  their  weapon 
systems  are  being  carried  out,  by  individual  System  Acquisition 
Managers  of  all  three  Services,  in  an  effort  to  provide 
interactive  and  electronically  displayed  Technical  Information. 


Ref  (1)  Fuller,  Joseph  J.,  Theodore  J.  Post,  and  Anne  S.  Mavor,  "Test 
and  Evaluation  of  the  Navy  Technical  Information  Presentation 
System  (NTIPS),  F-14A  Field  Test  Results,"  DTRC-88/036  (Sep 
1988) . 


Ref  (2)  Fuller,  Joseph  J. ,  Raymond  L.  LeBeau,  Anne  S.  Mavor,  Theodore 

J.  Post,  and  Charles  S.  Sawyer,  "Test  and  Evaluation  of  The 
Navy  Technical  Information  Presentation  System  (NTIPS),  AN/SPA- 
25D  Test  Results,"  DTRC-88/035  (Sep  1988). 

Ref  (3)  Thomas,  D.L.  and  J.D.  Clay,  "Computer-Based  Maintenance  Aids 
for  Technicians:  Project  Final  Report,"  Air  Force  Human 

Resources  Laboratory,  AFHRL-TR-87-44  (Aug  1988). 

Ref  (4)  Link,  W.R.,  J.C.  Von  Holle,  and  D.  Mason,  "Integrated 

Maintenance  Information  System  (IMIS):  A  Maintenance 

Information  Delivery  Concept,”  Air  Force  Human  Resources 
Laboratory,  AFHRL-TP-87-27  (Nov  1987). 
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1.2  DOD  AND  TRI-SERVICE  PROGRAMS  ESTABLISHED  IN  RESPONSE  TO 

TECHNICAL  INFORMATION  AUTOMATION  POLICY 

To  coordinate  and  standardize  the  increased  use  of 
computer-aided  logistic  support  throughout  the  three  Services, 
the  Department  of  Defense  established  the  Computer-aided 
Acquisition  and  Logistics  Support  (CALS)  program  [see  Ref  (5)], 
which  also  has  had  a  wide  effect  in  stimulating  progress  toward 
the  goal  of  TI  automation,  and  particularly  toward 
standardization  of  such  efforts. 


The  Department  of  Defense  established  [Ref  (5)],  and  later 
reiterated  [Ref  (6)],  a  policy  requiring  that  access  to  and 
the  delivery  of  system-related  logistic-support  information  be 
automated . 


For  example.  Ref  (6)  provided  the  following  directions: 

a.  For  systems  now  in  full-scale  development  or 
production,  program  managers  were  required  to  review 
specific  opportunities  for  cost  savings  or  quality 
improvements  that  could  result  from  changing  delivery 
or  access  using  the  Computer-aided  Acquisition  and 
Logistics  Support  standards. 

b.  For  systems  entering  development  after  September  1988, 
acquisition  plans,  solicitations,  and  related 
documents  required  specific  schedule  and  cost 
proposals  for: 


Ref  (5)  DEPSECDEF  MEMORANDUM  FOR  SECRETARIES  OF  THE  MILITARY 
DEPARTMENTS,  of  24  Sep  1985.  Sub j :  Computer  Aided  Logistic 

Support . 

Ref  (6)  DEPSECDEF  MEMORANDUM  FOR  SECRETARIES  OF  THE  MILITARY  DEPARTMENTS 
AND  DIRECTOR,  DEFENSE  LOGISTICS  AGENCY,  of  5  Aug  1988.  Sub  j : 
Computer-Aided  Acquisition  and  Logistic  Support. 
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(1)  integration  of  Contractor  Technical  Information 
systems  and  processes; 

(2)  authorized  Government  access  to  Contractor  data 
bases;  and 

(3)  delivery  of  Technical  Information  in  digital 
form. 

c.  DOD  components  were  to  program  for  automated  systems 
to  receive,  store,  distribute,  and  use  digital  weapon- 
system  Technical  Information,  including  achieving  the 
earliest  possible  date  for  digital  input  to  DOD 
engineering  data  repositories. 


More  recently,  the  Joint  Uniform  Service  Technical 
Information  System  (JUSTIS)  concept  has  been  announced,  a 
planned  effort  which  will  combine,  to  as  great  a  degree  as 
possible,  Tri-Service  procedures  and  equipment  for  acquisition 
and  control  of  system-support  Technical  Information. 


1.3  THE  INTERACTIVE  ELECTRONIC  TECHNICAL  MANUAL  CONCEPT 


The  culmination  of  this  effort  throughout  the  1980s  in 
response  to  the  DOD  policy  statements  cited  has  been  the 
development  '  of  the  Interactive  Electronic  Technical  Manual 
(IETM)  Concept.  The  IETM  Concept  involves  full  application  of 
existing  technological  capabilities  to  the  problems  of 
providing  Technical  Information  which  is  both  more  effective 
for  the  end  user  and  more  efficient  in  terms  of  acquisition, 
control,  and  update. 


The  IETM  Concept  involves  a  system  approach,  which  includes 
basically  all  of  the  following  components: 


6 


a.  A  standardized,  automated,  revisable  source  Data  Base. 

b.  Use  of  a  computer-controlled  authoring  system. 

c.  The  generation  of  digital  Technical  Information 
(containing  text  and  graphics) ,  either  directly  by  an 
Author,  or  automatically  by  computer.  This  Technical 
Information  is  recorded  on  an  electronically 
processible  medium  (optical  or  magnetic) ,  rather  than 
on  paper. 

d.  Technical  Information  (consisting  of  task-related 
increments)  which  is  optimally  arranged  and  formatted 
for  interactive  screen  presentation. 

e.  Presentation  (display  to  the  end  user)  by  means  of  a 
computer-controlled  Electronic  Display  System  (EDS) 
possessing  an  extensive  user-interaction  capability. 
The  EDS  is  capable  of  displaying  the  IETM,  performing 
related  logistic-support  functions,  and  interfacing 
with  other  Service  logistic-support  Information 
Systems . 


An  IETM  permits  a  user  to  locate  required  information  more 
easily,  and  to  present  it  faster,  more  comprehensibly,  more 
specifically  matched  to  the  configuration,  and  in  a  form  that 
requires  much  less  storage  than  paper.  Powerful 
troubleshooting  procedures  not  possible  with  paper  Technical 
Manuals  are  possible  using  the  computational  capability  of  the 
IETM  Display  Device. 


IETMs  will  be  used  by  maintenance  technicians,  afloat  and 
ashore;  to  maintain  and  operate  weapon  systems  by  Intermediate 
and  Depot  maintenance  activities;  and  by  training  personnel. 
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The  IETM  Concept  has  been  described  in  detail  in  Ref  (7)  . 


1.4  PREPARATION  OF  SPECIFICATIONS  AND  HANDBOOKS  FOR  SERVICE¬ 
WIDE  COORDINATION  OF  ACQUISITION  OF  AUTOMATED  TECHNICAL 

INFORMATION 

To  coordinate  this  wide-spread  effort,  the  Defense  Quality 
and  Standardization  Office  established  in  1989,  under  the  DOD 
Technical  Manual  Technology  Exchange  Subcommittee,  chartered  by 
DOD  INST  4151.9  [Ref  (8)],  an  Interactive  Electronic  Technical 
Manual  Working  Group,  chaired  by  the  Navy,  whose  primary 
functions  were  to: 

a.  Foster  the  exchange  of  ideas  and  the  agreement  on  a 

single  approach  regarding: 

(1)  the  acquisition  of  IETMs  which  use  computer 
technology  for  innovative  electronic  display; 
and 

(2)  presentation  of  Technical  Manual  Information 
among  all  Department  of  Defense  Agencies. 

b.  Develop  a  set  of  DOD  Specifications  for: 

(1)  The  acquisition  of  IETM  data;  and 


Ref  (7)  Rainey,  Samuel  C.,  Joseph  J.  Fuller,  and  Eric  L.  Jorgensen, 
"The  Electronic  Delivery  of  Automated  Technical  Information  for 
Logistics  Support  of  Navy  Weapons  Systems:  Potential,  System 
Description,  and  Status,"  DTRC-89/007  (Feb  1989). 

Ref  (8)  DoD  Instruction  4151.9  of  3  Jan  1989,  "DoD  Technical  Manual 
Program  Management . " 
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(2)  The  Electronic  Display  Systems  needed  for  the 
presentation  of  IETMs  for  the  maintenance  of  DOD 
weapons,  systems,  and  equipment. 


The  Working  Group  was  also  charged  with  the  responsibility 
of  providing  a  recommendation  to  the  DOD  CALS  Policy  Office 
concerning  inclusion  of  IETM  interchange  Specifications  into 
the  set  of  CALS  standards;  e.g.,  in  connection  with 
MIL-STD-1840 . 


The  Tri-Service  Working  Group  consists  of  representatives 
of  (a)  the  David  Taylor  Research  Center  (DTRC)  of  the  Navy, 
(b)  the  Air  Force  Logistics  Command  (AFLC-MMDE) ,  and  (c)  the 
US  Army  Communications-Electronics  Command  (AMCPM-TMDE) . 


With  DTRC  and  the  Air  Force  Human  Resources  Laboratory  (as 
an  advisor  to  AFLC)  contributing  the  primary  effort,  a  series 
of  five  Specifications  (see  Section  2.3)  and  Handbooks  for  IETM 
acquisition  has  been  drafted.  This  series  consists  of: 

■  A  Specification  governing  the  nature  of  the  Revisable 
IETM  Data  Base; 

■  A  Specification  providing  general  Content,  Style,  Format, 
and  User-Interaction  requirements  for  all  IETMs; 

■  A  Handbook  describing  for  a  System  Acquisition  Manager 
the  best  approach  to  writing  acquisition  Specifications 
for  individual  View  Packages  (to  be  used  for  IETM 
procurement) ; 

■  A  Handbook  presenting  requirements  for  the  Electronic 
Display  System; 
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■  A  Specification  presenting  requirements  for  an  IETM 
Quality  Assurance  Program. 

These  documents  have  been  widely  circulated  for  comment 
within  both  the  DOD  and  Industry 1 

These  drafts  were  also  developed  to  accomplish  as  near-term 
objectives  the  provision  of  a  suite  of  IETM  prototype 
acquisition  documents  for  use  by  major  DOD  programs  in 
establishing  initial  IETM  capabilities.  These  programs  include 
the  Navy's  A-12  Attack  Aircraft  Program,  the  Advanced  Tactical 
Fighter  Program  of  the  Air  Force,  and  the  M-l  Main  Battle  Tank 
Program  of  the  Army. 


1.5  PURPOSE  OF  PRESENT  REPORT 

The  purpose  of  the  present  Report  is  to  present  and  to 
describe  in  detail  one  of  these  draft  Specifications, 
specifically:  MI L-M- 1 ETMQA .  Quality  Assurance  (QA)  Program 
Requirements  for  Interactive  Electronic  Technical  Manuals 
(IETMs)  and  Associated  Technical  Information.  June  1990. 

A  series  of  four  other  Reports  has  been  prepared,  each 
Report  describing  one  member  of  the  set  of  five  acquisition 
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documents  prepared  by  this  Working  Group  [Ref  (9)  through 
Ref  (12)]. 


Section  2  of  this  Report  provides  an  overall  description  of 
this  suite  of  Acquisition  Specifications  and  Handbooks. 
Section  3  summarizes  the  Approach  and  Requirements  of  one  of 
the  five  documents;  in  this  case,  MIL-M-IETMQA. 


The  l-June-1990  draft  version  of  MIL-M-IETMQA  is  included 
in  this  Report  as  Appendix  A. 


Ref  (9)  Rainey,  Samuel  C.,  Eric  L.  Jorgensen,  and  Joseph  J.  Fuller, 
"Proposed  Draft  Military  Specification  for  Revisable  Data  Base 
for  Support  of  Interactive  Electronic  Technical  Manuals 
( IETMs ) , "  DTRC  Report  90/027  (Jul  1990). 

Ref  (10)  Jorgensen,  Eric  L.,  Samuel  C.  Rainey,  and  Joseph  J.  Fuller, 
"Proposed  Draft  Military  Specification  for  General  Content, 
Style,  Format,  and  User-Interaction  Requirements  for  Interactive 
Electronic  Technical  Manuals,"  DTRC  Report  90/028  (Jul  1990). 


Ref  (11)  Rainey,  Samuel  C.,  Eric  L.  Jorgensen,  and  Joseph  J.  Fuller, 
"Proposed  Draft  Military  Handbook  for  Preparation  of  View 
Packages  in  Support  of  Interactive  Electronic  Technical  Manuals. 
DTRC  Report  90/026  (Jul  1990). 


Ref  (12)  Jorgensen,  Eric  L. ,  Samuel  C.  Rainey,  and  Joseph  J.  Fuller, 
"Proposed  Draft  Military  Handbook  Presenting  Requirements  for  an 
Electronic  Display  System  for  Interactive  Electronic  Technical 
Manuals  (IETMs),"  DTRC  Report  90/025  (Jul  1990). 
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2.0  ACQUISITION  DOCUMENTATION  FOR 
INTERACTIVE  ELECTRONIC  TECHNICAL  MANUALS 
AND  ASSOCIATED  TECHNICAL  INFORMATION 


2.1  DEFINITIONS 


2.1.1  The  Interactive  Electronic  Technical  Manual  (IETM) . 

As  defined  by  the  Working  Group,  an  IBTH  is  a  Technical 
Manual,  prepared  (authored)  by  a  Contractor  and  delivered  to 
the  Government,  or  prepared  by  a  Government  activity,  in 
digital  form  on  a  suitable  medium,  by  means  of  an  automated 
authoring  system;  designed  for  electronic-screen  display  to  an 
end  user;  and  possessing  the  following  three  characteristics: 

a.  The  format  and  style  of  the  presented  information  are 
optimized  for  screen  presentation  to  assure  maximum 
comprehension;  that  is,  the  presentation  format  is  "frame- 
oriented",  not  "page-oriented". 

b.  The  elements  of  Technical  Information  constituting  the  TM 
are  so  interrelated  that  a  user's  access  to  the  information 
he  requires  is  facilitated  to  the  greatest  extent  possible, 
and  is  achievable  by  a  variety  of  paths. 

c.  The  computer-controlled  TM-Display  Device  can  function 
interactively  (as  a  result  of  user  requests  and  information 
input)  in  providing  procedural  guidance,  navigational 
directions,  and  supplemental  information;  and  also  in 
providing  assistance  in  carrying  out  logistic-support 
functions  supplemental  to  maintenance. 
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This  terminology  is  consistent  with  the  standard  DOD 
definition  of  Technical  Manual .  Ref  (8),  states: 

Technical  Manuals  are  publications  that  contain  instructions  for  the 
installation,  operation,  maintenance,  training,  and  support  of  weapon 
systems,  weapon-system  components,  and  support  equipment.  TM 
information  may  be  presented  in  any  form  or  characteristic,  including 
but  not  limited  to  hard  printed  copy,  audio  and  visual  displays, 
magnetic  tape,  discs,  and  other  electronic  devices.  They  normally 
include  operational  and  maintenance  instructions,  parts  lists  or  parts 
breakdowns,  and  related  technical  information  or  procedures  exclusive 
of  administrative  procedures.  Technical  Orders  (TOs)  that  meet  the 
criteria  of  this  definition  may  also  be  classified  as  TMs. 


2.1.2  The  View  Package. 

IETM  information,  as  provided  to  the  end  user  for  viewing 
on  an  Electronic  Display  Device,  will  be  constructed  in 
individual  task-oriented  increments  called  View  Packages. 


A  View  Package  (VP)  is  a  fully  organized  and  formatted  item 
of  computer-process ible  Technical  Information  derived  from  an 
IETM  Data  Base  and  capable  of  interactive  electronic  display  to 
an  end  user  by  means  of  an  Electronic  Display  System  (EDS)  .  In 
function  and  design,  a  View  Package  is  completely  equivalent  to 
an  individual  Interactive  Electronic  Technical  Manual.  A  View 
Package  may  be  constructed: 

a.  entirely  by  an  Author  using  an  automated  authoring 
system; 
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b.  completely  automatically  using  a  series  of  automated 
processes  (software)  which  perform  the  data-selection, 
structuring,  and  formatting  processes;  or 

c.  by  a  combination  of  the  above  two  approaches. 


A  View  Package  is  designed  to  support  a  specific  function 
in  the  operation  or  logistics-support  of  a  weapon  system  or 
other  military  equipment. 


2.1.3  Nature  and  Purpose  of  the  Revisable  IETM  Data  Base 

As  noted  above,  a  View  Package  is  created  entirely  from 
data  contained  in  a  Revisable  IETM  Data  Base  (IETMDB)  ,  which  is 
a  complete  collection  of  Data  Elements  relating  to  a  weapon 
system  or  other  equipment  acquired  by  the  Government  and 
constructed  in  a  standardized  procedure  in  order  to  provide  the 
following  capabilities: 

a.  Government  activities  or  DOD  Contractors  concerned 
with  logistic  support  for  the  weapon  system  involved 
can  access  the  Data  Base  directly  to  obtain  needed 
logistic-support  information  for  specific  purposes. 

b.  The  IETMDB  can  serve  as  the  basis  for  construction  and 
update  of  the  entire  suite  of  electronically  displayed 
interactive  weapon-system  Technical  Manuals  through 
the  use  of  automated  authoring  systems. 

c.  The  IETMDB  can  serve  as  the  basis  for  fully  automated 
construction,  by  either  a  Contractor  or  a  Government 
Activity,  of  View  Packages,  which  are  increments  of 
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interactive  electronically  presented  logistic-support 
Technical  Information. 

d.  Required  portions  of  the  IETMDB  can  be  interchanged  by 
means  of  standardized  procedures  throughout  the  DOD 
and  its  supporting  Contractors  on  a  real-time  basis 
when  needed. 


2.1.4  The  Electronic  Display  System  (EDS) 

The  EDS  is  a  computer-based  Technical  Information  system 
designed  to  accept,  process  and  integrate  Technical  Information 
for  prime-equipment  logistics  support,  and  display  that 
information  to  users.  The  EDS  is  also  intended  to  support 
inquiries  by  users  (in  addition  to  Operations  and  Maintenance 
users)  who  have  such  responsibilities  as  supply,  training, 
field-data  collection,  readiness  measurement,  operations 
scheduling,  maintenance  planning,  maintenance  quality  control, 
and  hardware  configuration  control.  The  software  supporting 
the  EDS  will  also  be  required  to  support  additional  (as  yet 
unspecified)  functions  in  the  future,  which  will  emerge  as 
technologies  and  standards  evolve.  Specifically,  the  EDS 
is  intended  for  use: 

a.  In  maintenance  Work  Centers  and  shops  to  support 
Troubleshooting  and  Planned  and  Corrective 
Maintenance; 

b.  In  portable  form  at  remotely  located  maintenance 
sites; 

c.  Embedded  in  a  weapon-system  control  panel  as  support 
both  for  System  operation  and  System  maintenance; 
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d.  In  presenting  operating  and  maintenance  information 
during  personnel  training  courses; 

e.  In  a  variety  of  centers  and  offices  in  support  of 
System-related,  logistics-supported  functions  which 
require  Technical  Information. 


The  Electronic  Display  System  will  consist  of  one  or  more 
computer-controlled  Devices  which  display  the  required 
Technical  Information  by  means  of  a  screen  (such  as  a  cathode- 
ray-tube  or  a  plasma  display)  either  in  a  pre-ordered  sequence 
or  in  random-access  increments,  as  called  for  by  the  user; 
e.g.,  a  maintenance  technician.  To  accomplish  this  display, 
the  IETM,  consisting  of  the  Technical  Information  recorded  on 
a  suitable  medium  (e.g.,  on  an  optical  disc) ,  is  designed  to  be 
loaded  into  the  EDS,  "read”  by  this  Device,  and  displayed  in  a 
sequence  as  directed  by  the  user. 


The  IETMs  to  be  used  by  this  Display  System  must 
accordingly  be  so  constructed  as  to  assure  full  compatibility 
with  the  operating  software  of  the  Display  Device,  and  must  be 
tested  by  the  preparing  Contractor  on  such  a  Display  Device 
prior  to  delivery. 


2.1.5  Summary 

As  noted,  all  IETMs: 

a.  Will  be  constructed  through  the  use  of  an  automated 
authoring  system,  and  will  consist  of  task-related 
increments  referred  to  as  View  Packages; 
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b.  Will  be  based  on  an  automated  system  Data  Base,  the 
IETMDB,  prepared  by  the  System  Prime  Contractor  for 
delivery  as  such  to  the  Government,  retention  for  his 
own  use,  or  both; 

c.  Will  consist  of  a  digital  data  stream  recorded  on  an 
optical  or  magnetic  medium,  but  not  paper, 
electronically  displayed  by  the  Electronic  Display 
System  in  terms  of  text  and  graphics; 

d.  Will  be  optimally  formatted  and  styled  for  screen 
presentation  (i.e.,  "frame  oriented"  rather  than 
"paper  oriented") . 

e.  Will  be  constructed  for  electronic  display  on  a  highly 
interactive  Electronic  Display  System,  which  will 
support  related  logistic-support  functions  and  which 
may  be  networked  for  interface  with  other  Service 
Information  Management  Systems. 


2.2  IETM  PROCUREMENT  OPTIONS 

Logistic-support  procedures  for  weapon  systems  and  related 
equipment  differ  to  some  extent  among  the  Services.  A  certain 
amount  of  necessary  variation  in  the  acquisition  procedures 
involving  the  VPs,  the  IETMDB,  and  the  EDS  has  been  provided  in 
the  system  of  Specifications  and  Acquisition  Handbooks 
developed  by  the  IETM  Working  Group. 


Thus,  these  Specifications  and  Handbooks  detail  several 
optional  approaches  in  the  acquisition  of  IETMs .  These  are  as 
follows: 
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a.  Using  appropriate  IETM  Specifications,  the  Service 
may  buy  whatever  directly-authored  Interactive 
Electronic  Technical  Manuals  are  required.  Although 
the  Author  (equipment  Prime  Contractor)  will  need  to 
establish  an  automated  equipment  or  weapon-system 
(source)  Data  Base,  this  Data  Base  will  not  be 
acquired  by  the  Government,  but  will  be  maintained  and 
used  by  the  Contractor,  both  for  the  preparation  of 
IETMs  and  for  other  purposes. 

(1)  As  an  option,  the  Government  might  contract  for 
on-line  access  to  technical  portions  of  this 
Contractor-owned  Data  Base.  In  such  a  case,  both 
content  and  accessibility  aspects  of  the  IETM 
Data  Base  would  have  to  be  constructed  to 
standard  requirements. 


b.  Acquisition  by  the  Government  of  directly  authored 
IETMs  (fully  prepared  and  validated  by  the  Contractor) 
as  well  as  the  TETM  Data  Base  upon  which  they  are 
based.  Government  acquisition  of  the  IETM  Data  Base 
may  involve  either  of  the  following  options: 

(1)  Delivery  to  the  Government  in  standardized  form 
and  subsequent  maintenance  by  the  Government 
(with  or  without  update  information  supplied  on  a 
continuing  basis  by  the  Contractor) ; 

(2)  Title  acquired  to  the  IETM  Data  Base  by  the 

Government,  but  with  the  Data  Base  retained  and 
maintained  in  the  Contractor's  plant.  The 

Government  to  be  provided  with  on-line  access  to 
the  Data  Base. 
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c.  Based  on  acquisition  of  the  IETM  Data  Base,  using 
either  option  b. (1)  or  b.(2),  preparation  of  View 
Packages  using  either  a  fully  automated  process  or  one 
which  is  essentially  fully  automated.  View  Packages 
could  be  prepared  either: 

(1)  By  the  Contractor  [based  on  Data-Base  acquisition 
option  b.(l)],  and  delivered  as  such  to  the 
Government,  or 

(2)  By  the  Government  [based  on  Data-Base  acquisition 
option  b. (2 )  ]  . 


2.3  SUMMARY  AND  PURPOSE  OF  THE  DRAFT  ACQUISITION 
SPECIFICATIONS  AND  HANDBOOKS  PREPARED  BY  THE  TRI-SERVICE 
IETM  WORKING  GROUP 

As  noted,  five  draft  Specifications  and  Handbooks  have 
been  prepared,  and  circulated  widely  for  DOD  and  Industry 
comment,  to  provide  System  Acquisition  Managers  with  the 
necessary  contractual  documentation  for  acquisition  of 
Interactive  Electronic  Technical  Manuals,  the  associated  source 
Data  Base,  and  the  necessary  Electronic  Display  Systems.  These 
statements  of  requirements  are  preliminary  and  will  certainly 
be  modified  as  experience  is  gained  with  the  acquisition, 
management,  and  use  of  this  type  of  Technical  Information,  as 
the  technology  advances,  and  as  the  Department  of  Defense 
improves  its  in-house  logistic-support  infrastructure  for 
support  of  IETMs. 


The  five  draft  Specifications  and  Handbooks  prepared  by 
the  Inter-Service  IETM  Working  Group  (of  which  Appendix  A  of 
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this  Report  is  one) ,  together  with  individual  statements  of  the 
purpose  of  each  document,  are  as  follows: 


2.3.1  The  Revisable  IETM  Data  Base  Specification 


2. 3. 1.1  Title 

Draft  MIL-D-IETMDB .  Reviseible  Data  Base  for  Support  of 
Interactive  Electronic  Technical  Manuals  (IETMs)  .  1  June  1990. 


2. 3. 1.2  Purpose 

This  Specification  contains  the  requirements  for  a 
Revisable  Interactive  Electronic  Technical  Manual  Data  Base 
(IETMDB)  to  be  constructed  by  a  weapon-system  Contractor.  This 
non-redundant  and  neutrally  formatted  Data  Base  is  intended  to 
be  the  single  source  of  data  for  all  Technical  Manuals  to  be 
used  in  support  of  a  given  weapon  system,  or  other  equipment 
being  acquired  by  the  Government.  This  Specification  may  be 
used  in  two  primary  modes: 

a.  as  a  set  of  standard  requirements  to  which  the 
Contractor  must  adhere  in  the  development  and 
maintenance  of  his  internal  Data  Base  for  subsequent 
conversion  to  Government-deliverable  form;  and 

b.  as  a  set  of  requirements  for  a  Data  Base  that  is 
physically  delivered  to  the  Government,  or  is 
maintained  by  the  Contractor  on  behalf  of  the 
Government . 
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2.3.2  The  IETM  General  Content.  Style,  Format,  and  User- 
Interaction  Requirements  Specification 


2. 3. 2.1  Title 

Draft  MIL-M-GCSFUI .  Manuals,  Interactive  Electronic 
Technical:  General  Content,  Style,  Format,  and  User- 
Interaction  Requirements  for.  1  June  1990. 


2. 3. 2. 2  Purpose 

This  Specification  contains  common  requirements  for  the 
Content,  Style,  Format,  and  User-Interaction  features  required 
for  Interactive  Electronic  Technical  Manuals  and  the  operating 
software  of  the  devices  upon  which  they  are  viewed.  These 
IETMs  are  to  be  delivered  to  the  Government  in  digital  form  and 
must  be  designed  for  interactive  display  to  the  maintenance- 
technician  end-user  by  means  of  a  computer-controlled 
Electronic  Display  System.  The  range  of  IETMs  for  which 
general  requirements .are  described  in  this  Specification  will 
cover  the  maintenance,  diagnostic,  training,  system-operation, 
parts-information,  and  installation  functions  which  are 
required  to  achieve  and  maintain  full  operational  capability  of 
a  specific  weapon  system  or  other  military  equipment. 


2.3.3  The  IETM  View  Package  Handbook 


2. 3. 3.1  Title 

Draft  MIL-HDBK-IETMVP .  Guidelines  for  Developing 
Specifications  for  Interactive  Electronic  Technical  Manual 
(IETM)  View  Packages.  1  June  1990. 
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2. 3. 3. 2  Purpose 


The  purpose  of  this  Handbook  is  to  provide  guidance  for 
the  preparation  of  individual  View-Package  Specifications,  so 
that  System  Acquisition  Managers  may  define  View  Package 
Requirements  quickly  and  effectively  for  the  numerous  different 
specialized  increments  of  Technical  Information  which  will  be 
required.  A  Handbook  of  this  type  has  been  referred  to  as  a 
meta-specification:  a  Specification  describing  how  to  write  a 
View  Package  Specification  which  is  the  end-item  Specification 
for  procurement  of  an  actual  IETM. 


2-3.4  The  IETM  OA  Program  Requirements  Specification 


2. 3. 4.1  Title 

Draft  MIL-M-IETMQA.  Quality  Assurance  (QA)  Program 
Requirements  for  Interactive  Electronic  Technical  Manuals 
(IETMs)  and  Associated  Technical  Information,  l  June  1990. 


2 . 3 . 4 . 2  Purpose 

This  Specification  prescribes  the  requirements  for  a 
Contractor's  QA  program  for  Interactive  Electronic  Technical 
Manuals  (IETMs)  and,  where  procured,  the  associated  IETM  Data 
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Bases  and  supporting  View  Packages.  The  requirements  herein 
cover  the  QA  process  and  present  the  plan  for  implementing  it, 
from  planning  through  final  submission  of  the  delivered  product 
for  acceptance;  they  apply  as  well  to  changes  and  revisions 
thereto. 


2.3.5  The  Electronic  Display  System  Handbook 


2. 3. 5.1  Title 

Draft  MIL-HDBK-EDS  (Navy) .  Electronic  Display  System  (EDS) 
for  Interactive  Electronic  Technical  Manuals  (IETMs) . 

1  June  1990. 

2. 3. 5. 2  Purpose 

This  Handbook  describes  the  basic  functional  requirements 
for  an  Electronic  Display  System  (EDS)  designed  to  display 
Interactive  Electronic  Technical  Manuals  (IETMs) .  It 
establishes  the  minimum  system  requirements  to  be  used  in  a 
detailed  Specification  for  competitive  procurement,  either  for 
portions  of  the  full  requirements  or  tailored  to  suit  the 
application,  user  environment,  device  compatibility,  and 
interfaces  to  existing  computer  systems. 

The  requirements  described  in  this  Handbook  are  of  three 
types: 


a.  Those  which  describe  the  Electronic  Display  System 
hardware ; 
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b.  Those  which  describe  the  EDS  software  of  the  display 
System  for  system  operation,  IETM  applications,  and 
utility  functions. 

c.  Those  which  specify  the  minimum  performance  of  the 
several  individual  Display  Devices  which  constitute 
the  EDS. 

To  awhieve  full  compatibility  of  the  EDS  with  the  IETMs 
and  View  Packages,  the  Display  System  Software  (as  well  as  the 
View  Package)  must  also  be  constructed  in  compliance  with 
MIL-M-GCSFUI . 


Each  of  the  three  Services  has  its  own  strategies  for 
developing  Specifications  and  Standards  for  an  Electronic 
Display  System.  This  Handbook  presents  the  existing  Navy 
concepts,  and  is  accordingly  identified  as  a  Navy-only 
document.  Proposed  concepts  of  the  other  Services  which  do  not 
differ  extensively  from  requirements  described  in  this  Handbook 
will  be  included  in  succeeding  versions  of  the  Handbook. 


2.4  RELATIONSHIP  OF  MIL-M-IETMQA  TO  OVERALL  SET  OF  IETM 

ACQUISITION  SPECIFICATIONS  AND  HANDBOOKS 

Each  of  the  other  four  Acquisition  Specifications  or 
Handbooks  in  the  IETM  series  provides  requirements  for  a 
specific  software  product  or  software/hardware  product  (the  EDS 
Handbook) .  It  is  the  function  of  MIL-M-IETMQA  to  provide  a 
contractual  document  to  assure  that  the  IETM  products  (IETMDB 
and  VPs)  are  of  high  quality  and  in  accordance  with  contractual 
requirements.  QA  procedures  described  by  MIL-M-IETMQA  do  not 
apply  to  the  Electronic  Display  System. 
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3.0  SUMMARY  OF  MIL-M-IETMQA, 

QUALITY  ASSURANCE  PROGRAM  REQUIREMENTS  FOR  IETMs 
AND  ASSOCIATED  TECHNICAL  INFORMATION 


3.1  QUALITY  ASSURANCE  PRINCIPLES  FOR  IETMS 

As  defined  in  connection  with  the  preparation  and  delivery 
of  Technical  Information,  Quality  Assurance  is  a  Contractor 
function.  In  general,  whether  the  TI  is  intended  for  use  on  a 
paper  medium,  or  constructed  for  interactive  display  as  in  the 
present  case,  the  Contractor's  QA  program  must  monitor  certain 
consignment  and  consecutive  processes  and  assure  that  they  are 
being  carried  out  in  a  professional  manner  so  as  to  provide  a 
comprehensible  quality  product  which  is  free  from  error.  These 
processes  are,  in  general: 

a.  Assembly  of  the  required  Technical  Source  Data  required 
for  system  support; 

b.  Establishment  of  the  necessary  staff,  automated 
authoring  system,  and  clearly  outlined  procedures  for 
generating  and  testing  the  TI; 

c.  Composing  and  reviewing  the  Technical  Information; 

d.  Validation  of  the  Technical  Information; 

e.  Correction  of  errors  found,  and  final  inspection; 

f.  Support  of  the  Government's  Verification  Process. 
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3 . 2  PHILOSOPHY  OF  QUALITY  ASSURANCE  PROCEDURES  AS  DESCRIBED  BY 

MIL-M-IETMQA 

The  approach  used  by  MIL-M-IETMQA  in  establishing 
requirements  for  IETM  Quality  Assurance  procedures  follows 
closely  that  of  the  existing  Quality  Assurance  Specifications 
MIL-Q-9858,  Quality  Program  Requirements  (which  applies  to 
hardware  equipment)  and  MIL-M-85357,  Manuals,  Technical: 
Quality  Assurance  Program,  Requirements  for  (which  applies  to 
paper  Technical  Manuals) . 


In  both  of  these  Specifications,  emphasis  is  given  to 
establishment  of  a  program  of  QA  surveillance  from  the 
initiation  of  the  equipment-manufacturing  or  TM-generation 
process  rather  than  establishment  of  inspection  procedures  late 
in  the  fabrication  sequence,  where  much  time  and  effort  may 
have  been  uselessly  applied  to  the  production  or  TM-generation 
process . 


3.3  DIFFERENCES  BETWEEN  QA  FOR  IETMS  AND  QA  FOR  PAPER 

TECHNICAL  MANUALS 

Although  the  philosophy  of  Quality  Assurance  for  Automated 
Technical  Information  is  identical  to  that  for  paper  Technical 
Information,  QA  procedures  for  IETMs  differ  from  those  for 
paper  Technical  Manuals  in  several  important  aspects: 

a.  Procedures  must  be  established  by  the  Contractor  for 
assurance  that  the  Source  Data  File,  the  IETMDB  itself, 
conforms  to  Government  Specifications  (particularly  as 
regards  standardized  access  and  interchange  qualities) ; 
that  it  is  complete;  and  that  it  is  free  from  error. 
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b.  Because  of  the  possible  use  of  highly  automated 
procedures  which  may  be  used  for  generating  View 
Packages,  similarly  automated  QA  methods  must  be 
developed  and  applied  to  assure  that  such  View  Packages 
are  complete,  comprehensible,  and  accurate. 

c.  QA  must  be  performed  using  a  specific  Electronic 
Display  System  approved  by  the  Government  to  assure 
that  the  Technical  Information  (and  its  interactive 
features)  looks  and  responds  to  the  Author  and  the  QA 
organization  in  the  same  way  as  it  will  to  the  end 
user. 

d.  The  entire  IETM  package  generated,  along  with  its 
auxiliary  and  interactive  features,  must  be  Validated 
as  a  system  to  assure  proper  display,  support  of 
related  functions  (e.g.,  maintenance  and  action 
reporting) ,  and  interfacing  with  other  networked  EDS 
units  and  Service  Information  Management  Systems. 

e.  IETM  QA  involves  control  and  Validation  of  certain 
software  processes  (e.g.,  a  View  Package  Compilation 
algorithm  or  applications  routines  incorporated  into 
the  IETM  package) .  Approved,  computer  programs  to 
accomplish  this  software  testing  will  be  required. 

f.  The  Contractor  will  be  required  to  generate  the  IETMs 
by  means  of  an  automated  authoring  system,  with 
sufficient  capability  to  carry  out  the  complex 
interactive  functions  required  by  the  IETM  suite  of 
Specifications  and  Handbooks. 


At  the  same  time,  the  computer-controlled  nature  of  IETMs 
offers  the  possibility,  far  more  than  in  the  case  of  paper  TI, 
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of  developing  computer  software  to  carry  out  extensive  QA  tests 
of  all  aspects  of  the  TI. 


3.4  THE  APPROACH  TO  QA  ADOPTED  BY  MIL-M-IETMQA 

The  Contractor's  QA  procedure  is  controlled  by 
MIL-M-IETMQA  through  the  requirement  that  the  Contractor 
prepare,  as  part  of  his  Prime  Equipment  Bid  Package,  a  Quality 
Assurance  Program  Plan  (QAPP) .  Once  approved  (accepted)  by  the 
Government,  this  QAPP  will  be  made  part  of  the  Contract  (part 
of  the  Contractual  Statement  of  Work) ,  and  will  be  used  as  the 
basis  for  continued  QA  inspections  by  the  Contractor's  QA 
organization,  which  must  be  separate  from  the  Contractor's 
Tl-generation  staff. 


Requirements  for  inspections  and  procedures  to  be  included 
by  the  Contractor  are  provided  in  detail  (in  14  areas)  by 
MIL-M-IETMQA  (Section  3.1.1  of  the  Specification). 


3 . 5  VALIDATION 

Central  to  the  entire  QA  process  is  the  emphasis  of 
MIL-M-IETMQA  on  a  careful  Validation  process.  It  requires 
preparation  of  a  Validation  Plan,  a  Prevalidation 
Certification,  and  Validation  of  the  IETM  (and  the  IETMDB) 
using  individuals  similar  in  training  and  experience  to 
intended  end  users  of  the  IETM.  Of  particular  importance  are 
the  requirements  for  Validation  of  the  automated  View  Package 
Compilation  Process,  when  such  a  process  is  used 
(Section  3.5.2),  and  for  Validation  of  the  entire  IETM  System 
(Section  3.5.3)  .  Validation  is  to  be  followed  by  correction  of 
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errors  found,  and  a  thorough  post-Validation  review 
(Section  3.5  of  the  Specification)  of  the  IETM. 


The  Validation  is  to  be  performed  using  an  EDS  identical 
to  the  equipment  which  will  ultimately  be  used  to  display  the 
IETM. 


3.5.1  Validation  of  IETM. 

Validation  of  IETM,  which  is  to  be  undertaken  only  after 
Contractor  certification  that  the  IETM  is  ready  for  Validation, 
is  to  be  performed  by  individuals  who  are  of  approximately  the 
same  education,  experience,  and  skill  level  as  the  actual 
target  audience  for  the  IETM.  The  Validator  must  use  a 
Government-specified  Electronic  Display  Device  to  establish  a 
display  mode  identical  to  that  involved  in  Fleet  use.  An 
operational  environment  must  be  used,  if  possible,  or 
simulated,  if  practicable. 


3.5.2  Validation  of  View-Packaae  Compilation  Process 

One  of  the  options  in  acquiring  Interactive  Electronic 
Technical  Manuals  involves  use  of  the  IETM  Data  Base  as  the 
source  of  Technical  Information  which  will  be  assembled 
automatically  (i.e.,  by  use  of  a  pre-prepared  computer 
program) ,  or  nearly  automatically,  into  individual  View 
Packages,  as  described  in  MIL-HDBK-IETMVP.  Contractor¬ 
generated  algorithms,  and  software  developed  for  preparation  of 
individual  View  Package  configurations  must  be  Validated. 


In  carrying  out  this  Validation,  the  Contractor  must 
demonstrate  the  capability  of  the  Contractor-generated  programs 
to  extract  appropriate  (defined)  information  entities  from  the 
IETMDB,  to  structure  them  into  coherent  procedural  or 
descriptive  Technical  Information,  and  to  format  this  TI  for 
display  to  the  end  user  using  a  Government-specified  or 
Government-furnished  Electronic  Display  Device. 

The  Validation  must  include  a  complete  exercise  of  all  TI 
sequences  (procedural,  descriptive,  or  fault-isolation) 
contained  in  the  View  Package;  i.e.,  the  Contractor  must 
Validate  the  compilation  process  by  Validating  the 
functionality  of  the  compiled  product. 


3.5.3  IETM  System  Validation 

In  addition  to  Validating  the  individual  tasks  and  non¬ 
procedural  data  of  the  entire  IETM,  the  Contractor  must  also 
Validate  the  entire  IETM  system,  with  its  various  utilities  and 
special  functions.  This  IETM  system  Validation  will  consist  of 
two  parts:  (1)  A  demonstration  of  the  system,  and  (2)  System 
software  tests. 

3. 5. 3.1  IETM  System  Demonstration 

The  Contractor  must  demonstrate  the  hardware  and  software 
accuracy,  capabilities,  and  compatibility  of  the  field  version 
of  the  IETM  system  with  Government  systems  and  contractual 
requirements.  Specifically,  the  IETM  system  demonstration 
must: 


a.  Be  performed  entirely  on  an  EDS  which  is  approved  by 
the  Government; 

b.  Demonstrate  the  system's  compliance  with  applicable 
Specifications  and  compatibility  with  Government 
systems ; 

c.  Demonstrate  maintenance  tasks,  displayed  for  a  selected 
sampling  of  the  various  subsystems  of  the  equipment 
involved,  in  various  levels  of  complexity; 

d.  Introduce  a  selected  sampling  of  simulated  faults  to 
exercise  the  retrieval  capabilities  (linking)  of  the 
system; 

e.  Demonstrate  all  system  utilities  and  special  functions, 
including  ad  hoc  user  access  to  data;  and 

f.  Demonstrate  the  general  visibility  and  readability  of 
the  IETM  data  as  displayed  on  the  Government-furnished 
EDS. 

3. 5. 3. 2  System  Software  Tests 

The  Contractor  must  develop  a  software  routine  capable  of 
exhaustively  testing  the  IETM  system.  This  program,  to  be  in 
accordance  with  MIL-STD-2167  and  MIL-STD-2168 ,  must,  as  a 
minimum: 


a.  Check  that  all  data  access  links  and  branches  are  valid 
and  operable; 

b.  Verify  that  there  are  no  "open  loops”  in  the  fault 
isolation  tasks  (i.e.,  that  each  task  ends  in  a  single 
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resolution,  and  each  retrieval  string  ends  in  isolation 
and  repair  of  a  discrete  fault) ; 

c.  Verify  that  there  are  no  unreferenced  or  inaccessible 
data  in  the  system;  and 

d.  Check  that  all  special  functions  are  fully  operable. 


3.5.4  Validation  of  the  IETM  Data  Base. 

When  required  by  the  Contract,  the  Contractor  must 
deliver,  as  an  adjunct  to  the  IETM,  an  IETM  Data  Base,  as 
described  in  MIL-M-IETMDB.  This  extensive  automated  Data  Base 
consists  of  (1)  a  wide  variety  of  information  entities  related 
to  the  weapon  system  (or  other  equipment) ;  (2)  a  series  of 
assigned  information-entity  attributes;  and  (3)  a  series  of 
defined  relationships  linking  the  information  entities. 
Validation  of  this  Data  Base  will  normally  represent  the  final 
action  in  the  QA  chain,  which  has  been  exercised  during  the 
assembly  of  the  Data  Base  from  source  data,  and  its 
organization  into  a  form  which  conforms  to  the  requirements  of 
MIL-M-IETMDB. 


3 . 6  REQUIREMENT  FOR  DESCRIPTION  OF  AUTOMATED  AUTHORING  SYSTEMS 

The  Specification  does  not  designate  use  of  any  specific 
type  of  IETM  authoring  system.  However,  it  requires  that  the 
Contractor's  Quality  Assurance  Program  Plan  provide  a 
description  of  the  automated  IETM  authoring  system  intended  for 
Contractor  use,  sufficient  to  demonstrate  to  the  Government  the 
capability  of  the  authoring  system  to: 
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a.  Support  Technical  Manual  Planning  and  Control. 

b.  Automatically  capture  data  from  a  wide  variety  of 
computerized  Data  Bases,  including  text,  graphics,  and 
tables . 

c.  Create  and  change  Technical  Information,  including 
text,  graphics  (both  vector  and  bit-map) ,  and  tabular 
data;  and  the  ability  to  merge  these  information 
elements  into  a  single  composite  element. 

d.  Provide  flexible  access  by  the  Author  to  files  in  the 
Data  Base  he  has  assembled  for  preparing  the  IETM  and 
any  associated  TI. 

e.  Provide  prompts  to  the  Author;  e.g.,  to  propose  proper 
sequences  of  writing,  to  identify  omissions  in  pre- 
established  plans,  and  to  identify  faulty  authoring 
logic. 

f.  Provide  the  Author  with  a  view  of  his  finished  product 
in  a  form  identical  to  that  in  which  it  will  be  viewed 
by  the  end  user;  (i.e.,  a  full  WYSIWYG  "What  You  See  Is 
What  You  Get"  capability) . 


To  accomplish  these  functions,  the  Contractor's  authoring 
system  will  be  required  to  contain  the  software  necessary  to 
provide  the  internal  linking  (providing  the  information-access 
sequences)  of  the  IETM  and  associated  IETM  Data  Base. 
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3 . 7  REQUIRED  COMPATIBILITY  BETWEEN  THE  IETM  AND  THE  ELECTRONIC 

DISPLAY  SYSTEM  (EDS) 

The  Specification  includes  the  requirements  that 
essentially  the  entire  Quality  Assurance  process  be  designed  to 
demonstrate  the  compatibility  of  the  IETMs  to  be  generated  with 
the  EDS  upon  which  the  end  user  will  display  the  Technical 
Information.  The  Contractor's  plan  for  obtaining  the  required 
Government-furnished  EDS  for  this  purpose  must  be  described  in 
the  QAPP  (Section  3.1.7  of  the  Specification).  The  appropriate 
EDS  is  thus  to  be  used  to  carry  out  Contractor  Quality  Reviews 
at  In  Process  Reviews,  and,  of  particular  importance,  during 
Validation. 


3.8  ADDITIONAL  CHARACTERISTICS  OF  MIL-M-IETMQA 

Through  enforcement  of  the  QAPP,  MIL-M-IETMQA  requires 
also  the  following: 

a.  Monitoring  to  assure  that  all  system  support  data 
sources  are  fully  utilized,  particularly  the  LSAR; 

b.  The  performance  of  Contractor  Quality  Reviews 
supplemented  by  Government-chaired  In-Process  Audits 
(IPAs)  ; 

c.  Control  of  Subcontractors  and  Vendors; 

d.  Preparation  of  a  Verification  support  plan. 


In  an  Appendix,  the  Specification  MIL-M-IETMQA  summarizes 
the  Government's  proposed  IETM  Acceptance  Tests  for  information 
purposes,  to  acquaint  an  IETM  Contractor  with  the  nature  of  the 
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inspection  which  his  IETM  (and  VPs  and  IETMBD)  must  pass  once 
it  is  delivered  to  the  Government. 
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APPENDIX  A 


Copy  of  Draft  Military  Specification  MIL-M-IETMQA.  Quality 
Assurance  (QA)  Program  Requirements  for  Interactive  Electronic 
Technical  Manuals  (IETMs)  and  Associated  Technical  Information . 
1  J  une  1990. 


Prior  to  the  publication  of  this  report  the  document  included  as 
Appendix  A  has  been  officially  submitted  to  the  DGD  Deii.u'L  Quality 
Standardization  Office  and  the  DOD  CALS  Policy  Office  by  the  Office  of  the 
Chief  of  Naval  Operations,  Code  403  -  Logistics  Policy  (OPNAV  LTR  4160  Ser 
403T/OU593  1  87  dtd  4  Jun  1990).  It  has  also  been  submitted  to  the  Pageless 
Technical  Manual  Working  Group  of  the  Aerospace  Industry  Association  for 
Review  and  Comment.  This  document  was  distributed  as  a  review  draft  and 
is  largely  a  DTRC  product  with  assistance  from  the  Air  Force  as  noted.  This 
Appendix  is  in  the  exact  form  that  was  submitted  to  these  organizations. 


MIL-M-IETMQA 
1  June  1990 


NOTE:  This  draft,  dated  30  May  1990,  prepared  by  DTRC/AFHRL/AFLC,  has  not 
been  approved  and  is  subject  to  change.  DO  NOT  USE  FOR  ACQUISITION 
PURPOSES . 


****REVIEW  DRAFT**** 

Beneficial  comments  (recommendations,  additions,  deletions)  and 
any  pertinent  data  which  may  be  of  use  in  improving  this 
document  should  be  addressed  to:  David  Taylor  Research  Center, 
Code  182.3,  Bethesda,  Maryland  20084-5000. 
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MILITARY  SPECIFICATION 


QUALITY  ASSURANCE  (QA)  PROGRAM  REQUIREMENTS  FOR 
INTERACTIVE  ELECTRONIC  TECHNICAL  MANUALS  ( IETMs ) 
AND  ASSOCIATED  TECHNICAL  INFORMATION 


1.  SCOPE 

1.1  Overview.  This  Specification  prescribes  the  requirements  for  a 
Contractor's  QA  program  for  Interactive  Electronic  Technical  Manuals 
(IETMs)  and,  whc.re  procured,  the  associated  IETM  Data  Bases  and  supporting 
View  Packages.  The  requirements  herein  cover  the  QA  process  from  planning 
through  final  submission  of  the  delivered  product  for  acceptance;  and  apply 
as  well  to  changes  and  revisions  thereto.  This  Specification  provides 
detailed  requirements  for: 

a.  Preparation  of  a  Quality  Assurance  Program  Plan  (QAPP); 

b.  Participation  in  a  Guidance  and  Quality  Planning  Conference; 

c.  In-Process  Audits  (IPA's)  of  products  and  processes  including 
compatibility  of  IETM's  with  a  Government-specified  Electronic 
Display  Device  (EDD);  this  compatibility  to  be  demonstrated 
during  IPAs  and  during  Validation; 

d.  Participation  in  Government-conducted  In-Process  Audits  (IPAs); 

e.  Validation  of  the  IETMs  and  related  Technical  Information; 

f.  Preparation  of  a  Verification  Support  Plan;  and 

g.  Retention  and  analysis  of  QA  records. 


1.2  Applicability.  This  Specification,  when  invoked  in  the  Contract  or 
other  contractual  document,  applies  to  a  given  IETM  procurement.  It  is  not 
intended  to  apply  to  procurement  of  commercial  and  off-the-shelf  Technical 
Manuals . 


1 . 3  Contractual  Intent. 


1.3.1  Contractor  Surveillance.  The  Contractor  shall  carry  out  his  QA 
Program  in  accordance  with  his  QAPP  as  submitted  to,  and  approved  by,  the 
Government,  and  incorporated  into  the  procurement  contract.  The 
Contractor's  QA  organization  shall  perform  surveillance  activities  to 
ensure  compliance  with  the  approved  QAPP,  applicable  IETM  specifications, 
and  other  contractual  documents.  These  activities  shall  commence  early  in 
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the  IETM-generation  process  and  continue  throughout  the  procurement.  (See 
3.3.3.  ) 


1.3.2  Government  Surveillance.  The  Contractor  shall  participate  in  a 
Guidance  and  Quality  Planning  Conference  to  be  conducted  by  the  Government 
(See  3.2),  and  in  a  series  of  Government-conducted  In-Process  Audits  (IPAs) 
at  scheduled  intervals  throughout  the  term  of  the  Contract.  (See  3.3.2.) 
The  purpose  of  these  IPAs  is  to  ensure  that  the  Contractor  is  complying 
with  the  approved  QAPP,  and  that  contractually  required  deliverables  meet 
applicable  IETM  specifications  and  other  contract  requirements. 


1.3.3  Responsibility  for  Inspection.  Unless  otherwise  specified  in  the 
Contract  or  order,  the  Contractor  is  responsible  for  the  performance  of  all 
inspections  as  specified  herein.  The  Contractor  shall  use  his  own 
facilities  or  any  commercial  laboratory  acceptable  to  the  Government.  The 
Government  reserves  the  right  to  perform  also  at  its  discretion  any  of  the 
inspections  set  forth  in  the  approved  QAPP,  applicable  IETM  specifications, 
and  other  contract  documents  where  such  inspections  are  deemed  desirable 
to  assure  that  supplies  and  services  conform  to  contract  requirements. 


1.4  Records  of  Information.  Records  of  QA  inspections  shall  be  kept  and 
made  available  to  the  Government  (See  3.2,  3.3,  3.4,  3.5). 


2.  APPLICABLE  DOCUMENTS 


2.1  Government  Documents.  The  following  documents  of  issue  in  effect  on 
the  date  of  invitation  for  bid  or  request  for  proposal  form  part  of  this 
Specification  to  the  extent  specified  herein.  Specification  and  Standard 
issues  shall  be  those  listed  in  that  issue  of  the  Department  of  Defense 
Index  of  Specifications  and  Standards  (DODISS)  specified  in  the  applicable 
contract. 

2.1.1  Specifications.  Standards  and  Handbooks. 


2. 1.1.1  Specifications . 

MIL-Q-9858  Quality  Program  Requirements 

MIL-M-GCSFUI  Manuals,  Interactive  Electronic  Technical:  General 

Content,  Style,  Format,  and  User  Interaction, 

Requirements  for 

MIL-M- IETMDB  Manuals,  Interactive  Electronic  Technical,  Revisable  Data 
Base  for  Support  of 
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2. 1.1. 2  Standards . 

MIL-STD-109  Quality  Assurance  Terms  and  Definitions. 

MIL-STD-1388-2A  Logistic  Support  Analysis  Record,  DoD 

Requirements  for 

MIL-STD-1840  Automated  Interchange  of  Technical  Information 
DOD-STD-2167  Defense  System  Software  Development 
DOD-STD-2168  Defense  System  Software  Quality  Program 

2. 1.1. 3  Other  Government  Documents. 

MIL-HDBK-EDS  <navyj  Electronic  Display  System  for  (Navy)  Interactive 

Electronic  Technical  Manuals. 

MIL-HDBK-IETMVP  Preparation  of  View  Packages  in  Support  of 

Interactive  Electronic  Technical  Manuals 

2 . 2  Other  Documents. 

2.3  Order  of  Precedence.  In  the  event  of  a  conflict  between  the  text  of 
this  Specification  and  the  references  cited  herein,  the  text  of  this 
Specification  shall  take  precedence. 


3.  REQUIREMENTS 


3.1  IETM  Quality  Assurance  Program.  The  Contractor  shall  establish  a  QA 
program  in  accordance  with  the  requirements  of  this  specification,  which 
supplements  and  details  the  requirements  of  MIL-Q-9858,  to  ensure  the 
development  of  technically  accurate  and  complete  IETMs.  Terminology  shall 
follow  MIL-STD-109,  except  where  superseded  by  the  terminology  of  this 
Specification.  Quality  Assurance  procedures  for  software,  developed  in 
accordance  with  DOD-STD-2167  and  delivered  as  part  of  the  IETM  and 
associated  Technical  Information  (for  example,  the  IETM  Data  Base  and 
supporting  View  Packages  where  ordered)  shall  conform  to  the  requirements 
of  DOD-STD-2168.  The  Contractor's  QA  program  shall  encompass  the 
accountability  for,  and  development  of,  QA  functions  required  for  the 
management  of  the  following  IETM  program  elements: 

a.  Source  data; 

b.  Intermediate  products,  including  IPAs; 
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c . 

Graphics ; 

d. 

System  software; 

e . 

Database  integrity; 

f . 

Authoring  system; 

g- 

Validation; 

h. 

Records ; 

i . 

Contractor  operating 

procedures ; 

j- 

Verification  support; 

and 

k. 

Final  product. 

3.1.1  IETM  Quality  Assurance  Program  Plan  (QAPP).  The  Contractor  shall 
prepare  a  Quality  Assurance  Program  Plan  (QAPP)  in  accordance  with  the 
requirements  of  this  Specification.  The  QAPP  shall  describe  the  scope  and 
approach  of  the  Contractor's  IETM  QA  Program.  The  QAPP  shall  include: 

a.  A  list  of  all  contractually  deliverable  IETM  items. 

b.  QA  Structure  and  Organization  and  its  functional  relationship 
to  the  IETM-Generation  Organization. 

c.  Description  of  the  Authoring  system. 

d.  Word  Lists 

e.  Describe  a  plan  for  obtaining  and  using  a  Government-specified 
or  Government-furnished  Electronic  Display  Device  and  associated 
View  Package  if  required  for  Quality  Reviews,  prevalidation,  and 
Validation. 

f.  A  listing  of  the  Contractor's  written  operating  procedures 

g.  A  listing  of  audit  procedures  for  quality  reviews 

h.  A  listing  of  critical  operations  involved  in  IETM  generation. 


i.  Audit  procedures  for  each  IETM  deliverable 

j .  Subcontractor  and  Vendor  control  Plan 

k.  Sampling  inspection  plan 

l.  Description  of  computer-supported  inspection  and  validation 
procedures  using  the  Government-approved  Electronic 
Delivery/Display  Device  (EDD) 
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m.  Validation  Plan 

n.  Verification  Support  Plan 


3.1.2  Implementation  of  OAPP.  The  Contractor  shall  submit  the  QAPP  to  the 
Government  for  approval.  The  Government  will  furnish  written  notice  of  the 
acceptability  of  the  Contractor's  QAPP.  The  QA  Program,  as  approved  by  the 
Government  through  review  of  the  QAPP,  shall  be  fully  implemented  by  the 
Contractor.  This  implementation  shall  be  evidenced  by  establishment  of  the 
following: 

a.  Comprehensive  Contractor  operating  instructions  for  all 
procedures  related  to  development  of  IETMs ; 

b.  In-Process  Audit  (IPA)  procedures  and  records; 

c.  Data  and  system  software  validation  procedures; 

d.  Corrective  and  preventive  action  procedures;  and 

e.  Management  support  of  the  QA  program. 


3.1.3  Identification  of  Contractually  Deliverable  Items.  The  first  part 
of  the  QAPP  identifies  the  scope  of  the  QA.  effort  by  identifying  all 
contractually  deliverable  items,  the  Contract  Specifications  with  which 
they  must  comply,  and  the  schedule  for  their  inspection  and  delivery.  The 
QAPP  shall  list: 

a.  All  interim  and  final  deliverables  in  complete  correspondence 
with  the  Contract  Data  Requirements  list  (CDRL)  of  the  Contract. 

b.  Specifications  with  which  each  deliverable  item  listed  in  3.1.1 
a.  must  comply  in  order  to  be  in  complete  correspondence  with 
the  Contract. 

c.  A  preliminary  schedule  for  each  deliverable  item  listed  in 
complete  correspondence  with  the  Contract;  any  schedule  changes 
(early  or  late  accomplishment)  anticipated  by  the  Contractor 
shall  be  indicated.  The  schedule  for  each  IETM  deliverable  item 
shall  include  the  following: 

(1)  Conduct  of  Government  IPAs  (including  review  of  interim 

deliverables ) . 

(2)  Inspection^ ) . 

(3)  Validation. 

(4)  TM  Readiness  for  Verification  (proposed  start  of 

Verification) . 

(5)  Estimated  Date  of  Readiness  for  Delivery. 

(6)  Any  other  Contractually  Required  Milestone. 
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3.1.4  QA  Structure  and  Organization.  The  Contractor's  QA  organization 
shall  have  the  responsibility  for  implementation,  administration,  and 
conduct  of  the  QA  program.  It  shall  function  as  the  point  of  contact 
between  the  Contractor  and  the  Government  in  all  matters  related  to  QA. 


3. 1.4.1  Functional  Relationships.  The  Contractor  shall  describe  the 
structure  and  functional  relationships  of  the  QA  organization. 


3. 1.4. 2  Personnel .  The  Contractor's  organization  shall  be  staffed  by  an 
adequate  number  of  QA  personnel  qualified  to  implement  their 
responsibilities.  The  Quality  Assurance  personnel  shall  be  independent 
from  the  IETM-generation  personnel.  (This  requirement  does  not  intend  to 
suggest,  however,  that  fulfillment  of  the  requirements  of  this 
Specification  is  the  responsibility  of  any  specific  Contractor 
organization,  function,  or  person.)  The  Contractor's  IETM  Quality 
Assurance  personnel  shall  have  well-defined  responsibilities;  authority  to 
identify  and  evaluate  Quality  Assurance  problems;  and  methods  to  initiate, 
recommend,  or  provide  solutions  and  corrective  actions  as  indicated.  The 
QAPP  shall: 


a.  Contain  a  schedule  which  shows  the  estimated  number  of  personnel 
per  month  that  the  Contractor  will  use  to  accomplish  the  Quality 
Assurance  tasks  under  the  Contract; 

b.  Explain  how  the  Contractor's  QA  personnel  are  organized  to 
retain  their  independence  from  personnel  responsible  for  IETM 
generation,  so  as  to  be  able  to  carry  out  their  QA 
responsibilities  in  an  objective  manner. 


3.1.5  Description  of  the  Authoring  System.  The  QAPP  shall  state  that  the 
Authoring  System  contains,  as  a  minimum,  the  following  features: 

a.  Capability  to  support  Technical  Manual  Planning  and  Control. 

b.  Automated  capturing  of  data  from  a  wide  variety  of  computerized 
data  bases,  including  text,  graphics,  and  tables. 

c.  Ability  to  create  and  change  Technical  Information,  including 
text,  graphics  (both  vector  and  bit  map),  and  tabular  data;  and 
the  ability  to  merge  these  information  elements  into  a  single 
composite  element. 

d.  Highly  flexible  access  by  the  Author  to  files  in  the  data  base 
he  has  assembled  for  preparing  the  IETM  and  any  associated  TI. 

e.  Software  necessary  to  provide  the  internal  linking  (providing 
the  information-access  sequences)  of  the  IETM  and  associated 
IETM  Data  Base. 
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f.  Ability  to  provide  prompts  to  the  Author;  e.g.,  to  propose 
proper  sequences  of  writing,  to  identify  omissions  in  pre- 
established  plans,  and  to  identify  faulty  authoring  logic. 

g.  Capability  to  provide  the  Author  with  a  view  of  his  finished 
product  in  a  form  identical  to  that  in  which  it  will  be  viewed 
by  the  end  user;  (i.e.,  a  full  WYSIWYG  "What  You  See  Is  What  You 
Get"  capability). 


3.1.6  Word  List.  The  contractor  shall  describe  in  the  QAPP  the 
Contractor's  approach  to,  and  identify  software  for  comparison  of,  authored 
text  with  a  Government-approved  word  list  (See  3. 1.8. 6);  and  describe  the 
Contractor's  plan  for  identifying  and  obtaining  Government  approval  for  use 
of  words  considered  necessary  but  which  are  not  on  the  Government-approved 
word  list.  The  QAPP  shall  also  include  the  Contractor's  recommendations 
for  the  "system-specific"  word  list,  which  will  supplement  the  "Common"  and 
"Technical  Area"  word  lists. 


3.1.7  Display  Device.  The  Contractor  shall  describe  a  plan  for  obtaining 
and  using  a  Government-specified  or  Government-furnished  Electronic  Display 
Device  and  associated  View  Package,  if  required,  for  Quality  Reviews, 
Prevalidation,  and  Validation.  Government  specifications  for  the  display 
device  include  both  hardware  and  software. 


3.1.8  Written  Operating  Procedures .  Writers,  editors,  illustrators, 
typists,  vendors,  and  others  at  all  stages  of  IETM  generation  shall  operate 
in  accordance  with  written  procedures.  These  procedures  shall  be  approved 
by  the  QA  organization  prior  to  their  implementation.  Portions  of  these 
procedures  which  relate  directly  to  product  quality  shall  be  cited  in  the 
QAPP.  Such  procedures  shall  be  originated,  revised,  and  controlled  within 
the  framework  of  the  overall  Quality  Assurance  Program,  and  shall  be 
reviewed,  evaluated,  and  up-dated  as  required  to  assure  that  they  reflect 
current  practices  and  that  compliance  with  these  procedures  results  in 
production  of  a  quality  product. 

3. 1.8.1  Style  Guides.  In-house  style  guides  provide  written  standards  for 
Technical  Manual  -personnel  and  are  required  for  the  generation  of  well- 
written  TMs .  If  these  guides  already  exist  in  a  form  adaptable  to  contract 
needs  (i.e.,  in  a  form  which  supports  Technical  Manual  creation  for 
interactive  electronic  display,  not  paper  pages),  they  may  be  used  either 
as-is  or  with  only  minor  modifications,  provided  also  that  they  are 
consistent  with  the  Format  and  Style  Requirements  expressed  in 
MIL-M-GCSFUI ,  which  may  be  used  in  place  of  existing  company  style  guides. 
In-house  or  other  style  guides  the  Contractor  proposes  to  use  shall  be 
cited  in  the  QAPP. 

3. 1.8. 2  Procedures  List.  The  QAPP  shall  list  (by  name,  number,  and 
date)  the  Contractor's  written  procedures  for  the  generation  of  IETM 
information;  at  a  minimum,  the  Contractor  shall  have  written  procedures  in 
the  following  areas: 
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(1)  Source-data  control 

(2)  Inspection  of  deliverables 

(3)  Validation  of  procedural  and  non-procedural  Technical 

Information  (TI) 

(4)  Critical-operation  inspections 

(5)  Contract  changes 

(6)  Evaluation,  review,  and  update  of  organizational  operating 

procedures 

(7)  Maintenance  of  Quality  Assurance  records 

(8)  Government  correspondence  control 

3. 1.8. 3  Source  Data.  The  QAPP  shall  describe  plans  to  ensure  that  the 
most  current  source  data  are  available  and  utilized  for  IETM  generation 
(See  3. 1.8.2).  The  following  are  examples  of  the  types  of  items  and 
control  documents  that  are  considered  appropriate  Technical  Information 
(TI)  source  data: 


Description  of  Source  Data 


(1)  Logistics  Support  Analysis  Record  (LSAR) 

(2)  Procurement  and  test  specifications 

(3)  Technical  descriptions  contained  in  project  proposal 

(4)  Photos  of  mockups  or  equipment 

(5)  Support-equipment  description  data 

(6)  Task-analysis  data 

(7)  Maintenance  concepts 

(8)  Special  user-personnel  qualifications 

(9)  Notes  and  materials  from  vendors 

(10)  Failure  Modes  and  Effects  Analysis  ( FMEA )  data 

(11)  Engineering  reports 

( 12 )  Blueprints/drawings/sketches 

(13)  Vendor  brochures  and  commercial  manuals 

(14)  Engineering  Change  Proposals  (ECPs) 

(15)  Maintenance  Engineering  Analysis  Records  (MEARs) 

(16)  System  and  Subsystem 

(17)  Support  Equipment  Requirement  Sheets  (SERS) 

(18)  Provisioning  Parts  Breakdowns  (PPBs) 

(19)  Development  hardware  location  and  availability 

(20)  Classification  data  (DD254) 

(21)  Local  Engineering  Specifications  (LES) 

(22)  Manual  Change  Releases  (MCR) 

(23)  Technical  Information  Deficiency  and  Evaluation  Reports 

(TIDERs) 

(24)  Validation  results 

(25)  Verification  results 

(26)  Design  Changes  Notices  (DCN) 

(27)  Supply  Item  Change  Requests  (SICR) 

(28)  Advance  Change  Notices  (ACNs) 


3. 1.8. 4  LSA/LSAR .  The  QAPP  shall  specifically  include  a  plan  to  assure 
that  direct  (on-line)  communication  exists  between  the  LSAR  and  the  IETM 
authoring  system,  and  that  the  LSAR  is  carefully  followed  in  constructing 
the  IETM. 
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3. 1.8. 5  Drawings .  The  QAPP  shall  include  measures  proposed  by  the 
Contractor  to  assure  the  effective  use  of  design  and  manufacturing  drawings 
(e.g.,  level  III  drawings)  for  the  IETM  generation  without  the  necessity 
to  completely  redraft  the  drawings. 

3. 1.8. 6  Word  Lists.  The  QAPP  shall  incorporate  a  listing  of  the  approved 
Government  Word  Lists  to  be  used  by  the  Contractor  in  generation  of  the 
IETM. 


3.1.9  Contractor  Quality  Reviews.  Many  operations  in  the  IETM-production 
process  require  inspection  so  as  to  discover  and  correct  any  TI  defects. 
In  these  critical  operations,  it  is  least  expensive  to  correct  defects  at 
an  early  phase,  rather  than  later  in  the  IETM-generation  process.  For 
example,  proofreading  and  typographical-error  correction  are  most  cost- 
effectively  accomplished  immediately  upon  completion  of  text  authoring. 
(These  processes  may  be  simplified  by  incorporation  of  automated  spell¬ 
checkers,  permissible  word  lists,  and  automated  syntax-control  routines 
into  the  automated  authoring  process.)  Similarly,  early  examination  of 
pencil  sketches  or  proposed  layouts,  and  early  correction  of  identified 
errors,  is  cost  effective  than  making  corrections  to  the  final  ink  drawing. 
Accordingly: 

a.  The  QAPP  shall  list  the  Contractor's  written  quality  review 
procedures  by  name,  number,  and  date.  At  a  minimum,  the  list  shall 
contain  written,  critical-operation  inspection  procedures  in  the 
following  areas: 

( 1 )  Proofreading 

( 2 )  Check  of  prepared  graphics  versus  the  writer ' s  input 

(3)  Editorial  review  of  author's  text-graphics  module  layouts. 

b.  The  Contractor's  quality  review  procedures  shall  contain  records 

which  include  the  name  of  the  review,  review  date,  name  and  number 
of  the  item  prepared,  method  used  the  correct  the  material,  the 
name  of  the  inspector  (proofreader,  editor,  etc.),  and 

certification  that  all  errors  found  were  corrected. 

c.  The  QAPP  shall  contain  procedures  which  will  define  how  the 
contractor  assures: 

( 1 )  Maintenance  procedures  shall  have  all  input  conditions 

explicitly  identified.  The  steps  contained  in  the  maintenance 
procedures  shall  use  concise  text  and  simple  graphics  to 
instruct  the  user.  Tasks  shall  be  cross-referenced  with 
appropriate  descriptive,  troubleshooting,  parts,  and 

operational  information. 

(2)  Steps  within  a  maintenance  procedure  shall  be  presented 
sequentially  and  in  their  entirety.  Information  that  is  not 
applicable  because  of  information  track  or  weapon  system 
configuration  shall  not  be  displayed.  All  warnings,  notes  and 
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cautions  associated  with  a  maintenance  action  shall  be  presented 
so  that  the  user  is  aware  of  all  potentially  dangerous  actions. 
Maintenance  procedures  shall  be  presented  as  the  technical 
writer  originally  intended  them  to  be  presented. 


3.1.10  Critical  Operation  Inspection.  Critical  operation  inspections  are 
IPAs  conducted  early  in  the  development  process  when  QA  processes  are  most 
beneficial  and  cost-effective.  For  example,  a  prevalidation  inspection  of 
IETM  data  could  allow  detection  and  relatively  easy  correction  of  problems 
which  might  otherwise  cause  the  termination  of  a  validation  procedure.  The 
QAPP  shall  incorporate  a  provision  for  a  final  (pre-delivery)  review,  as 
described  in  Section  3.5.  The  QAPP  shall  identify  the  Contractor's 
critical  operation  inspections,  including  the  phase  to  which  they  will  be 
applied,  and  the  intent  and  scope  of  the  inspections. 


3.1.11  Audits  for  IETM  deliverables.  The  contractor  shall  describe  the 
audit  procedures  for  each  IETM  deliverable  to  ensure  compliance  with  its 
contractual  specifications.  The  contractor  shall  also  explain  procedures 
for  maintaining  records  of  defects  found  and  corrective  actions  taken. 


3.1.12  Control  of  Subcontractors  and  Vendors.  The  Contractor  shall 
describe  the  approach  by  which  he  will  ensure  that  IETM  products  prepared 
by  subcontractors  and  vendors  satisfy  all  the  contractual  quality 
requirements.  The  Contractor  may:  (1)  require  that  the  subcontractor  or 
vendor  prepare  and  follow  a  QAPP  using  this  Specification;  (2)  perform  a 
receiving  inspection  to  ensure  that  work  accomplished  under  the 
subcontractor's  own  QA  plan  meets  the  requirements  of  this  Specification; 
or  (3)  establish  a  procedure  to  transform  vendor  input  into  a  form  which 
meets  Contract  requirements.  The  Contractor's  QAPP  shall  describe  the 
approach  by  which  he  will  ensure  compliance  of  subcontractor  or  vendor 
products  with  contractual  quality  requirements. 


3.1.13  Sampling  Inspection  Plan.  The  contractor  shall  describe  the 
overall  sampling  inspection  plan. 


3.1.14  Electronic  Deliverv/Displav  Device  ( EDD )  Validation  Procedures. 
The  contractor  shall  describe  the  computer-supported  inspection  and 
validation  procedures  using  the  Government- approved  Electronic 
Delivery/Display  Device. 


3.1.15  Validation  Plan.  The  Contractor  shall  prepare  a  Validation  Plan 
which  covers  each  IETM  item  and  details  the  anticipated  schedule, 
requirements,  and  procedures  for  performing  this  Quality  Assurance 
function.  This  Validation  Plan  is  an  interim  deliverable,  part  of  the 
QAPP,  and  forms  part  of  the  contract  requirements. 
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3.1.16  Verification  Support  Plan.  The  Contractor  shall  develop  a 
verification  support  plan,  describing  the  Contractor's  general  approach  to 
supporting  a  Government-conducted  verification.  The  plan  shall  be  reviewed 
and  approved  by  the  Government.  The  Contractor  shall  prepare  a  supplement 
to  the  verification  plan,  detailing  the  plan  for  providing  contractor 
support  of  the  verification  effort. 


3.2  Guidance  and  Quality  Planning  Conference.  A  Guidance  and  Quality 
Planning  Conference,  convened  by  the  Contractor,  will  be  conducted  by  the 
Government  TM  Logistics  Element  Manager  at  the  start  of  work  to  review  the 
Contractor's  understanding  of  applicable  specifications,  formal 
instructions,  established  policies,  and  program  requirements. 


3.2.1  Schedule.  The  Conference  shall  be  held  within  45  days  of  contract 
award.  Attendance  should  include  all  Government  and  Contractor  personnel 
who  will  be  involved  in  the  specific  QA  program.  Anticipated  Quality 
Assurance  problems  will  be  identified  and  resolved  at  this  Conference,  by 
either  the  Contractor  or  Government. 


3.2.2  QAPP .  The  Contractor  shall  explain  his  QAPP  at  the  Quality 
Assurance  Planning  Conference,  showing: 

(1)  how  the  QAPP  meets  all  of  the  requirements  of  this  Specification 
(Section  3.1);  and 

(2)  how  the  Quality  Assurance  Plan  will  function  to  assure  a  quality 
product  in  accordance  with  Contract  Specifications. 


3.3  In-Process  Audits  (IPAs).  Audits  shall  be  conducted  to  enable  the 
Contractor's  Quality  Assurance  personnel  or  the  Government  to  verify  that 
the  QAPP  is  being  properly  followed,  product  quality  reviews  are  being 
correctly  implemented,  defects  are  being  identified,  and  appropriate 
corrective  actions  are  being  taken. 


3.3.1  Initial  Government  IPA.  During  the  initial  (Government)  IPA,  the 
Contractor  shall  provide  specific  demonstrations  of  the  following: 

a.  The  use  of  the  authoring  system  to  be  employed  in  generating  the 
IETM,  including  a  demonstration  that  the  authoring-system  functions 
described  in  the  QAPP  are  in  working  order  and  are  being  used  in 
the  IETM-generation  process.  Relationship  of  the  IETM-authoring 
process  to  acquisition  and  use  of  relevant  source  data 
(particularly  the  LSAR)  shall  be  specifically  addressed. 

b.  Use  of  the  approved  Government-specified  or  Government- furnished 
Electronic  Display  System  which  the  Contractor  intends  to  use  for 
Quality  Assurance  Checks,  for  Validation,  and  for  Final-Draft 
Inspection. 
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c.  The  approved-word- list  comparison  routines  which  the  Contractor 
intends  to  use  during  the  IETM-generation  process.  The  three  Word 
Lists  themselves  ("Common",  "Technological  Area",  and  "System 
Specific")  shall  also  be  demonstrated. 


3.3.2  Government  Audits.  The  Contractor  shall  make  available  to  the 
Government  its  Quality  Assurance  records  for  interim  and  final 
deliverables,  at  scheduled  In-Process  Audits  (IPAs)  to  enable  the 
Government  to  ensure  that  the  QAPP  is  both  implemented  and  effective.  The 
Contractor  shall  also  support  the  Government-conducted  IPAs  by  providing 
access  to  finished  deliverables,  validated  Technical  Information,  and  other 
quality-assurance  records  as  specified  in  the  QAPP.  Access  to  Contractor¬ 
generated  TI  material  shall  be  provided  for  Government  audit  by  means  of 
the  approved  Government-specified  Electronic  Display  Device  cited  in 
Section  1.1  and  described  in  the  QAPP. 


3. 3. 2.1  Location.  Unless  otherwise  specified.  Government  IPAs  will  be 
conducted  at  the  Contractor's  facility.  The  IPAs  will  be  chaired  by  the 
Government  IETM  QA  representative.  At  the  IPA,  the  Contractor  shall 
present  the  Contractor's  Quality  Assurance  records  for  Government  audit  to 
ensure:  (1)  that  required  inspections  are  being  carried  out  in  accordance 
with  the  QAPP:  (2)  that  all  defects  discovered  by  the  inspections  were 
recorded  and  corrected;  and  (3)  that  all  contractually  required 
deliverables  are  being  inspected  and  meet  contractual  standards. 


3. 3. 2. 2  Validation  Listing.  Government  personnel  shall  be  provided  with 
a  complete  listing  of  all  Technical  Information  which  has  been  Validated 
prior  to  the  commencement  of  the  IPA.  Validated  TI  shall  prove  to  be 
logical,  complete,  safe,  and  technically  accurate.  Validated  procedural 
TI  must  lead  to  successful  accomplishment  of  the  procedures  by  personnel 
who  are  representative  of  the  intended  users,  with  the  information 
presented  by  means  of  a  Government-specified  Electronic  Display  Device. 
Audited  Validation  records  shall  accurately  record  the  Validation  efforts. 
(See  3.4.3  through  3.4.6) 


3. 3. 2. 3  Inspections .  The  Government  reserves  the  right  to  make  any 
inspection  it  deems  necessary  to  assure  that  IETM  delivered  is  of  high 
quality.  Government  participation  in  the  Quality  Assurance  process 
(including  inspection  and  acceptance  of  the  final  product)  will  generally 
be  exercised  by  means  of  Audits  of  the  Contractor's  Quality  Assurance 
actions  to  ensure  that  the  required  inspections  have  been  properly  made. 
This  shifts  the  emphasis  from  previously  conducted  In  Process  Reviews 
(IPRs)  in  which  Government  Inspection  involved  detailed  assessment  of 
product  quality  at  an  intermediate  level  of  production,  to  a  procedure 
which  focuses  Government  attention  on  evaluation  of  the  effectiveness  of 
the  Contractor's  QA  procedures  as  described  in  the  QAPP. 
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3.3.3  Contractor  Audits.  The  Contractor  auditing  process  is  a  continuous 
effort  designed  to  ensure  that  Contractor  personnel  responsible  for  TI 
generation  are  following  the  QAPP  and  applicable  work  instructions.  Any 
deviations  from  the  QAPP  shall  be  explained  in  the  Contractor  audit  records 
and  corrected  if  they  represent  a  hindrance  to  meeting  contractual 
requirements . 


3. 3. 3.1  Contractor  Audit  Plan.  The  Contractor  is  required  to  include  in 
his  QAPP  a  plan  for  auditing  operating  procedures ,  to  assure  proper 
performance  of  all  actions  described  in  the  QAPP,  and  required  by  Sections 

3.1  through  3.5  of  this  Specification. 


3. 3. 3. 2  Audit  Records .  The  Contractor's  auditing  personnel  shall  maintain 
records  documenting  the  periodic  audits  made.  These  records  shall  include: 

(1)  Name  of  the  operation  audited 

(2)  Date 

(3)  Name  of  auditors 

(4)  Defects  found 

(5)  Method  of  defect  correction  or  explanation  for  non-correction 

of  defect 

(6)  Name  of  person  making  correction 

(7)  Name  of  person  certifying  that  operation  has  been  so  modified, 

so  as  to  be  in  accordance  with  the  corrected  written  operating 
procedures . 


3.4  Validation.  The  objective  of  Validation  is  to  ensure  that  the 
Contractor  has  provided  accurate  and  adequate  IETMs ,  IETM  Data  Base,  and 
required  View  Packages  (if  not  provided  by  the  government)  for  support  of 
the  equipment,  in  accordance  with  the  approved  Validation  Plan.  Validation 
shall  be  accomplished  on  all  deliverables  and  changes  thereto.  The  overall 
Validation  process  shall  include  the  following  steps: 

(1)  Preparation  of  a  Validation  Plan; 

(2)  Prevalidation  Certification  of  TI; 

(3)  Validation; 

(4)  Validation  of  the  IETM  Data  Base; 

(5)  Validation  of  View-Package  Compilation  Process; 

(6)  IETM  System  Validation. 


3.4.1  Validation  Plan.  The  Contractor  shall  prepare  a  Validation  Plan 
which  covers  each  IETM  item  and  details  the  anticipated  schedule, 
requirements,  and  procedures  for  performing  this  Quality  Assurance 
function.  This  Validation  Plan  is  an  interim  deliverable,  part  of  the 
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QAPP,  and  forms  part  of  the  contract  requirements.  Specifically,  a 
Validation  Plan  shall  contain  the  following  items: 

(1)  Titles  and  identification  numbers  of  all  deliverables  to  be 
Validated 

(2)  Schedules  for  Validation 

(3)  Cognizant  Contractor  organization  and  personnel  responsible  for 
accomplishing  the  Validation  effort 

(4)  Site  locations,  support  equipment,  facilities,  test  equipment, 
materials,  and  tools  required  during  Validation 

(5)  General  characteristics  relating  to  skill  level  of  target 
audience  personnel  to  be  used  in  Validation  of  procedural  TI. 

(6)  Next  higher  assembly ( ies ) /system( s )  required  to  support  the 
effort  (e.g.,  if  a  procedure  entails  installing  a  black  box  on 
an  aircraft,  the  aircraft  or  a  mock-up  of  one  is  the  next  higher 
assembly) 

( 7 )  Associated  TI  recommended  for  concurrent  or  consecutive 
Validation 

(8)  Special  safety  precautions 

( 9 )  Any  special  environmental  requirements 

(10)  Record-keeping  system  to  be  used  in  Validation  of  both 
procedural  and  non-procedural  Technical  Information  (See 
paragraphs  3.4.3  through  3.4.5.) 

(11)  How  Validation  of  procedures  is  to  be  accomplished 

(12)  A  Fault-Simulation  list  for  use  in  Validation  of 
Troubleshooting  procedures.  (See  3. 4. 3. 3) 

(13)  Approach  to  evaluation  of  usability  of  the  IETM  as  generated 
by  the  Contractor;  specifically,  establishment  of  the  adequacy 
and-  accuracy  of  information-access  procedures  established  by 
the  author,  and  identification  of  any  man-machine  interface 
problems  involving  effective  use  of  the  IETM  on  the 
Government-specified  Electronic  Display  Device. 

(14)  Application  of  results  of  Validation  to  the  correction  of 
deficiencies . 


3.4.2  Prevalidation  Certification.  A  Contractor's  authorized  technical 
representative  shall  inspect  (i.e.,  "Prevalidate")  the  TI  to  ensure  that 
it  is  complete,  based  on  his  knowledge  and  judgment.  Based  on  his  review, 
the  Contractor  shall  certify  that  the  TI  permits  efficient  performance  <->f 
the  intended  equipment-support  functions  for  which  the  TI  is  designed. 
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Furthermore,  the  Contractor's  authorized  technical  expert  shall  certify 
that  the  TI  is  safe,  logical,  and  technically  accurate. 


3. 4. 2.1  Prevalidation  Records.  The  Contractor  shall  maintain 
Prevalidation  records  which  include:  the  name  of  the  authorized  technical 
representative,  the  date  of  Certification,  and  name,  number,  and 
configuration  of  the  deliverable  item.  The  records  shall  also  include  the 
Certification  by  the  authorized  Contractor  technical  representative  that 
each  individual  procedure  is : 

(1) Safe.  Danger  to  a  user  in  the  field  is  minimized  when 
performing  all  procedures  as  written. 

(2)  Complete.  All  steps  necessary  to  the  performance  of  the 
individual  procedure  have  been  included  and  the  procedure 
reflects  the  mission  capability  of  the  equipment. 

(3)  Logical.  The  procedure  is  in  an  order  which  is  the  most 
efficient  for  completion  of  the  maintenance  or  operation  task. 

( 4 )  Technically  Accurate .  All  operating  procedures ,  fault-isolation 
procedures,  check-out  procedures,  alignment  procedures, 
calibration  procedures,  and  system-descriptive  statements,  etc., 
are  free  of  errors . 

(5)  Written  in  words  included  in  the  provided  Word  List,  or 
otherwise  approved  by  the  Government. 


3. 4. 2. 2  Certification  Form.  The  Contractor  shall  execute  a  Certification 
Form  attesting  to  the  adequacy  and  accuracy  of  the  TI  as  stated  above.  The 
Certification  shall  be  signed  by  an  authorized  Contractor  representative. 
Individual  errors  noted' during  this  process  shall  be  corrected  prior  to 
Validation;  alternatively,  the  whole  TI  item  may  be  returned  to  the  Author 
for  improvement. 


3.4.3  Validation  of  IETM.  Validation  of  IETM  shall  be  undertaken  only 
after  the  Contractor's  authorized  technical  representative  has  certified 
that  the  IETM  is  ready  for  Validation.  The  validation  shall  be  performed 
by  individuals  who  are  of  approximately  the  same  education,  experience,  and 
skill  level  as  the  actual  target  audience  for  the  IETM.  The  Validator 
shall  follow  exactly  the  procedures  set  forth  in  the  IETM  and  shall  use  a 
Government-specified  Electronic  Display  Device  to  establish  a  display  mode 
identical  to  that  involved  in  Fleet  use.  An  operational  environment  shall 
be  used,  if  possible,  or  simulated,  if  practicable. 


3. 4. 3.1  Validation  of  Procedural  TI.  Contractor  personnel  shall  Validate 
each  step  in  every  procedure  using  a  Display  Device  of  the  same  type  as 
that  which  will  be  used  to  display  the  IETM  to  the  end  user.  The  procedure 
shall  be  presented  in  final  form  exactly  as  it  will  be  displayed  to  the  end 
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user,  with  the  same  user-interface  capabilities  (i.e.,  input-output 
functions,  special  functions). 


3.4.3. 1.1  Validation.  The  Validator  shall  be  able  to  perform  the 
procedure  without  error.  If  for  any  reason  a  Validator  fails  to  perform 
a  procedure  without  error,  the  procedure  shall  be  carefully  analyzed  for 
(a)  technical  errors  and  (b)  incomprehensibility.  If  technical  errors  are 
found,  corrections  shall  be  made  or  alternative  accurate  TI  shall  be 
prepared  and  re-Validated.  If  portions  of  the  TI  are  incomprehensible  to 
the  initial  target-audience  Validator,  the  procedure  may  be  accepted  if 
four  additional  target-audience  Validators  perform  the  procedure  without 
error.  If  the  procedure  is  reworked,  the  material  shall  be  re-Validated 
with  a  new  Validator.  Minor  corrections  (which  must  be  cited)  may  be  made 
without  re-Validation. 


3. 4. 3. 1.2  Validation  Records.  The  Contractor  shall  keep  Validation 
records  for  each  item  of  Procedural  TI.  These  records  shall  include  the 
name  of  the  Validator,  his  educational  level,  experience,  and  position 
classification,  as  well  as  the  date,  signature,  identifying  number,  and 
configuration  of  the  procedural  TI  Validated;  method  of  correction  used; 
and  the  signatures  of  both  the  person  who  made  the  correction  (or  other 
change)  and  the  inspector  who  certified  that  all  errors  found  were 
corrected.  The  records  shall  further  include  a  copy  of  the  procedure  after 
any  errors  detected  during  Validation  have  been  corrected  and  incorporated, 
and  the  procedure  successfully  Validated.  Hand-annotated  corrections  are 
acceptable  providing  that  they  are  fully  legible  and  understandable  by  the 
Validator.  When  corrections  are  extensive,  the  material  shall  be  reworked 
prior  to  re-certification  and  Revalidation. 


3. 4. 3. 2  Validation  of  Non-Procedural  TI.  The  Validation  of  TI  items  which 
are  not  of  a  procedural  nature,  such  as  Descriptive  Text,  wire  lists, 
schematics,  IPB  information,  and  the  like,  shall  be  accomplished  by 
comparing  them  to  the  actual  hardware  or  to  applicable  source  data 
(reports,  engineering  drawings,  etc.).  This  comparison  shall  be 
accomplished  by  Contractor  personnel  at  the  Contractor's  facility,  using 
a  Display  Device  identical  to  that  upon  which  the  IETM  will  be  displayed 
to  the  final  user. 


3. 4. 3. 2.1  Validation.  All  non-procedural  TI  shall  be  Validated  against 
the  actual  equipment  when  possible,  or  in  unusual  cases  against  source 
data,  and  shall  correspond  to  it  in  all  respects.  If  source  data  are  used 
in  the  Validation  process,  the  Contractor  shall  ensure  that  they  correspond 
to  the  actual  equipment.  All  defects  (errors)  shall  be  corrected. 
Validation  of  descriptive  TI,  where  some  basis  of  comparison  is  used  other 
than  the  actual  equipment,  shall  be  approved  by  the  Government. 


3. 4. 3. 2. 2  Validation  Records.  Validation  records  for  each  item  of  non¬ 
procedural  TI  shall  be  kept  by  the  Contractor.  These  records  shall  include 
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the  signature  of  the  Validator;  date;  name,  number,  and  configuration  of 
the  TI;  method  of  correction  used;  and  the  signatures  of  both  the  person 
who  made  the  correction  and  the  inspector  who  certified  that  all  errors 
found  were  corrected. 


3. 4. 3. 3  Validation  of  Troubleshooting  Procedures.  Validation  of  major 
established  branches  of  the  Troubleshooting  procedures  shall  be  performed 
with  the  weapon-system  hardware  to  which  the  Troubleshooting  TI  applies. 
This  part  of  the  validation  effort,  shall  be  performed  in  conjunction  with 
(1)  tabletop  comparison  with  source  data  to  ensure  the  accuracy  and 
consistency  of  fault  isolation  paths,  and  (2)  the  IETM  System  Validation 
(see  3.4.6).  This  effort  shall  be  planned  and  carried  out  so  as  to 
establish  a  high  degree  of  confidence  in  the  IETM  troubleshooting 
procedures  as  a  whole. 


3. 4. 3. 3.1  Troubleshooting  Procedures  List.  The  Contractor  shall  prepare 
a  list  of  troubleshooting  branches  to  be  validated,  and  a  list  of  proposed 
fault  simulations  to  be  used  during  Validation  of  Troubleshooting 
procedures.  The  list  will  include  a  wide  variety  of  faults.  It  will  be 
part  of  the  Validation  Plan  of  the  QAPP.  When  the  Validation  Plan  is 
approved,  the  list  will  be  used  to  select  faults  to  permit  Validation  of 
the  Troubleshooting  TI.  The  Fault-Simulation  List  shall  contain: 

a.  Fault  symptoms 

b.  Fault  simulated 

c.  Fault  to  be  inserted 
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3. 4. 3. 3. 2  Example  List.  The  following  is  an  example  Fault  Simulation 
List.  Requirements  cited  under  Section  3. 4. 3.1,  Validation  of  Procedural 
TI,  shall  apply. 


SYMPTOM 

FAULT  SIMULATED 

FAULT  INSERTED 

(1) 

Hydraulic  pump 

Defective  turret-power  Disconnect  wires 

fails  to  run  or 

relay  box 

from  relay  on  C/B 

makes  abnormal 

in  turret-power 

noise 

relay  box 

(2) 

Hydraulic  pump 

Defective  pump  motor 

Open  connection  to 

fails  to  run  or 

brushes  in  pump 

makes  abnormal 

motor . 

noise 

Defective  pressure 

Disconnect  pressure 

switch 

switch . 

Low  fluid  level  in 

Reduce  fluid  level 

reservoir 

in  reservoir  below 
normal  level. 

(3) 

System  pressure 

Defective  pressure 

Install  defective 

is  incorrect 

switch 

pressure  switch 

4.4 

Validation  of  the 

IETM  Data  Base.  When 

required  by  the  Contract 

the  Contractor  shall  provide,  as  an  adjunct  to  the  IETM,  an  IETM  Data  Base, 
as  described  in  MIL-M-IETMDB .  This  extensive  automated  data  base  consists 
of  a  wide  variety  of  information  elements  related  to  the  weapon-system  (or 
other  equipment),  a  series  of  assigned  information  element  attributes,  and 
a  series  of  defined  relationships  linking  the  information  elements. 
Validation  of  this  data  base  will  normally  represent  the  final  action  in 
the  QA  chain,  which  has  been  exercised  during  the  assembly  of  the  Data 
Base  from  source  data,  and  its  organization  into  a  form  which  conforms  to 
the  requirements  of  MIL-M-IETMDB. 


3.4.*. 1  Validation  Procedures.  Validation  of  the  IETMDB  shall  be  carried 
out  procedurally  (in  accordance  with  3.4)  in  the  same  fashion  as  the 
Validation  of  the  several  types  of  TI  included  in  the  IETM.  Specifically, 
Validation  of  the  IETMDB  shall  establish  that  the  Contractor  has 
successfully  met  the  following  four  overall  requirements  in  establishing 
the  File: 

(a)  The  IETMDB  meets  all  specific  requirements  of  MIL-M-IETMDB. 

(b)  All  information  contained  in  the  IETMDB  is  technically  accurate, 
based  on  comparison  of  the  data  elements  with  actual  system 
hardware,  or  (where  approved  by  the  Government)  with  source  data. 
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(c)  All  relationships  (established  links),  as  defined  in  the  Data 
Element  Dictionary  of  the  IETM  Data  Base,  are  in  place  and 
operable . 

(d)  The  IETMDB  is  capable  of  being  used,  in  accordance  with  MIL-HDBK- 
IETMVP,  to  construct  individual  View  Packages  to  support  the  Weapon 
System.  (See  3.4.5.) 


3. 4. 4. 1.1  Human  Participation.  Requirements  (a),  (c),  and  (d),  and  to 
some  extent  (b),  lend  themselves  to  automated  (computer-based)  processes 
as  the  most  effective  method  of  assuring  that  the  IETMDB  meets  the 
requirements  cited.  It  is  not  intended  that  the  IETMDB  Validation 
procedure  be  entirely  automated;  human  participation  in  Validation  is 
considered  necessary,  specifically  in  satisfying  requirement  (b)  above. 


3. 4. 4. 1.2  QAPP .  The  Contactor  shall  describe  in  the  QAPP  the  technical 
approach  proposed  for  Validation  of  this  material.  The  approach  shall 
fully  differentiate  between  computer-supported  technical  procedures,  and 
those  to  be  accomplished  by  humans. 


3. 4. 4. 1.3  Revalidation .  When  the  IETMDB  is  changed  in  any  way  after 
Validation,  the  changed  material  and  other  portions  of  the  IETMDB  affected 
shall  be  re-Validated .  This  process  shall  assure  that  all  linkages 
previously  established  are  still  operable  and  valid;  specifically,  that  no 
effect  on  the  accuracy  of  any  related  information  has  been  adversely 
affected  by  the  change. 


3.4.5  Validation  of  View-Package  Compilation  Process.  One  of  the  options 
involved  in  acquiring  Interactive  Electronic  Technical  Manuals  involves  use 
of  the  IETM  Data  Base  as  the  source  of  Technical  Information  which  will  be 
assembled  automatically,  or  nearly  automatically,  into  individual  View 
Packages,  as  described  in  MIL-HDBK-IETMVP .  If  a  Government  View  Package 
is  not  provided,  the  Contractor-generated  algorithms  and  software  for 
individual  View-Package  configurations  shall  be  Validated. 


3.4.5. 1  Demonstration.  The  Contractor  shall  provide  complete 
demonstration  of  the  capability  of  the  Contractor-generated  programs  to 
extract  appropriate  (defined)  information  elements  from  the  IETMDB,  to 
compose  their  elements  into  coherent  procedural  or  descriptive  Technical 
Information,  and  to  format  this  .TI  for  display  to  the  end  user  using  a 
Government-specified  or  Government-furnished  Electronic  Display  Device. 
The  Validation  procedure  shall  include  a  complete  exercise  of  all  TI 
sequences  (procedural,  descriptive,  or  fault-isolation)  contained  in  the 
View  Package;  i.e.,  validate  the  compilation  process  by  validating  the 
functionality  of  the  compiled  product. 


HIL-M-IETMQA 


19 


1  Juna  1990 


MIL-M-IETMQA 
1  June  1990 


3. 4. 5. 2  Validation  Procedure.  The  QAPP  shall  contain  the  Contractor's 
proposed  procedure  for  accomplishing  this  critical  function.  (The  QAPP 
shall  show  the  means  of  establishing  the  technical  accuracy  of  the  actual 
data  used  in  Validating  the  IETMDB ;  once  this  accuracy  is  established,  it 
need  not  be  re-established  during  Validation  of  the  View-Package 
compilation  process.) 


3.4.6  IETM  System  Validation.  In  addition  to  validation  of  the  individual 
tasks  and  non-procedural  data,  the  IETM  system,  with  its  various  utilities 
and  special  functions,  shall  be  validated.  The  IETM  system  validation 
shall  be  performed  prior  to  the  delivery  of  IETM  data  to  the  Government, 
and  shall  consist  of  two  parts:  (1)  a  demonstration  of  the  system,  and  (2) 
system  software  tests. 


3. 4. 6.1  IETM  System  Demonstration.  The  field  version  of  the  IETM  system 
hardware  and  software  shall  be  demonstrated  to  the  extent  required  to 
establish  confidence  in  its  accuracy,  capabilities,  and  compatibility  with 
Government  systems  and  contractual  requirements.  The  extent  of  the 
demonstration  shall  be  in  accordance  with  a  sampling  plan  included  in  the 
Contractor's  Validation  Plan.  Specifically,  the  IETM  system  demonstration 
shall : 

a.  Be  performed  entirely  on  an  EDD,  as  specified  and  approved  by  the 
Government ; 

b.  Demonstrate  the  system's  compliance  with  applicable  specifications 
and  compatibility  with  Government  systems; 

c.  Demonstrate  maintenance  tasks,  displayed  for  a  selected  sampling  of 
the  various  aircraft  systems,  in  various  levels  of  complexity; 

d.  Introduce  a  selected  sampling  of  simulated  faults  to  exercise  the 
retrieval  capabilities  (linking)  of  the  system; 

e.  Demonstrate  all  system  utilities  and  special  functions,  including 
ad  hoc  user  access  to  data;  and 

f.  Demonstrate  the  general  visibility  and  readability  of  the  IETM  data 
as  displayed  on  the  Government-furnished  EDD. 


3. 4. 6. 2  System  Software  Tests.  The  Contractor  shall  develop  a  software 
routine  capable  of  exhaustively  testing  the  IETM  system.  This  program 
shall  be  developed  in  accordance  with  MIL-STD-2167  and  MIL-STD-2168 ,  and 
shall,  as  a  minimum: 

a.  Check  that  all  data  access  links  and  branches  are  valid  and 
operable; 
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b.  Verify  that  there  are  no  "open  loops"  in  the  fault  isolation  tasks 
(i.e.,  that  each  task  ends  in  a  single  resolution,  and  each 
retrieval  string  ends  in  isolation  and  repair  of  a  discrete  fault); 

c.  Verify  that  there  are  no  unreferenced  or  inaccessible  data  in  the 
system;  and 

d.  Check  that  all  special  functions  are  fully  operable. 


3. 4. 6. 3  Support  Equipment.  Government  approved  support  equipment  shall 
be  utilized  in  the  performance  of  validation.  Simulation  or  substitution 
of  support  equipment  shall  be  approved  by  the  Government.  It  is  the 
responsibility  of  the  Contractor  to  obtain  Government  furnished  equipment 
in  order  to  support  the  validation  effort. 


3.5  Post-Validation  Review.  The  Contractor  shall  perform  a  post¬ 
validation  review.  This  review  does  not  apply  to  the  IETM  Data  Base. 


3.5.1  Discrepancies .  This  review  shall  assure  that  all  discrepancies 
noted  during  validation  have  been  addressed  prior  to  the  certification  and 
acceptance  of  the  IETM.  This  includes  review  of  remedial  actions  taken  by 
the  IETM  preparation  group  in  response  to  the  discrepancies  recorded  in  the 
applicable  validation  records. 

3.5.2  Records .  Contractor  personnel  shall  ensure  that  all  applicable 

inspector's  records  cited  under  Section  1.4  are  available  prior  to  the 
post-validation  review  and  that  they  indicate  full  compliance  with 

Specifications.  Contractor  personnel  shall  ensure  that  every  procedure  on 
the  task  list  is  included  in  the  TI,  that  all  procedures  have  both  a 
Prevalidation  Certification  and  target-audience  Validation  record,  and  that 
the  final  draft  of  all'  procedural  TI  conforms  exactly  to  the  Validation 
records.  Specifically: 

a.  Validation  records  for  all  non-procedural  TI  shall  show  that  all 
the  TI  is  in  conformance  with  the  associated  equipment. 

b.  Documentation  shall  be  presented  to  show  that  the  final  drafts  of 

all  IETM  items  are  in  exact  compliance  with,  all  Contract 
Specifications,  including  Format,  Style,  and  User-Interaction 

Specifications.  (See  3.1.3  b.) 

c.  Records  shall  be  maintained  of  the  inspections,  the  signatures  of 

the  inspectors,  identifications  of  the  deliverables  inspected, 
discrepancies  found,  if  any,  and  Certifications  that  the 
deliverable  fully  meets  Contract  Specification  requirements.  Any 
deviations  shall  be  explained  in  writing  by  the  Contractor's 
authorized  technical  authority. 
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3.6  Verification  Support.  Verification  shall  be  performed  on  validated 
IETM's  under  the  jurisdiction  of  the  Government,  and  may  include  Contractor 
support.  Verification  of  tasks  and  system  operation  shall  be  conducted  on 
the  Government- furnished  EDD  and  supporting  systems.  Verification  shall 
be  conducted  to: 

a.  ensure  that  the  IETM  data  is  suitable  for  the  intended  maintenance 
environment; 

b.  ensure  that  it  is  usable  by  its  intended  audience;  and 

c.  ensure  its  compatibility  with  Government  systems. 


3.6.1  Verification  Procedures.  Verification  is  accomplished  by  qualified 
personnel,  of  the  prescribed  skill  level,  from  the  operating  command  or 
facility  assigned  to  operate  and  maintain  the  weapon  system  to  which  the 
IETM  data  applies.  The  Government  will  perform  final  acceptance  tests  on 
the  IETM  data  subsequent  to  approval  of  changes  incorporated  into  the  data 
as  a  result  of  verification  findings. 


3.6.2  Verification  Support  Plan.  As  part  of  the  QAPP,  the  Contractor 
shall  develop  a  verification  support  plan,  describing  the  Contractor's 
general  approach  to  supporting  a  Government-conducted  verification.  The 
plan  shall  be  reviewed  and  approved  by  the  Government.  The  Contractor 
shall  prepare  a  supplement  to  the  verification  plan,  detailing  the  plan  for 
providing  contractor  support  of  the  verification  effort.  Contractor  support 
of  verification  shall  consist  of  the  following: 

a.  Technical  support,  as  required; 

b.  Assistance  in  performing  verification  procedures; 

c.  Recording  and  disposition  of  verification  comments;  and 

d.  Certification  that  all  discrepancies  and  defects  noted  during 
verification  have  been  corrected  or  resolved. 


3.7  Sampling.  Sampling  of  TI  is  authorized  for  In-Process  Audits  but  is 
not  permitted  for  Validation  nor  for  the  Post-Validation  Review.  Any 
sampling  plan  prepared  by  the  Contractor,  for  IPAs  or  any  other  critical- 
operation  audits,  shall  be  cited  in  the  QAPP  and  approved  by  the 
Government . 


4 .  Definition  of  Terms. 


Acceptance  Test 

The  final  test  series  performed  by  the  Government  on  the  IETM, 
following  Verification.  If  the  IETM  is  considered  satisfactory  as 
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a  result  of  these  tests  (meets  all  contractual  requirements),  it 
will  be  accepted  by  the  Government. 

Final  Draft 

The  version  of  the  complete  IETM  submitted  for  final  inspection  by 
the  Contractor's  QA  organization  after  corrections  noted  during 
Validation  (of  procedural,  non-procedural,  and  fault-isolation  TI) 
have  been  incorporated,  but  before  Verification. 

In-Process  Audit  (IPA) 

A  review  of  contractual  status,  on-going  QA  efforts,  interim 
deliverables,  or  draft  TI  under  preparation,  which  may  be  carried 
out  at  any  time  during  the  IETM  development,  to  permit  the 
Government,  to  the  level  desired,  to: 

(1)  assess  the  extent  to  which  the  Contractor  is  following  the  QAPP 
in  the  conduct  of  his  QA  program  and  to  evaluate  the  success  of 
this  program; 

(2)  ensure  that  the  technical-requirements,  documentation,  and 
Technical  Information  are  being  developed  according  to  the 
applicable  Specifications  and  the  QAPP; 

(3)  ensure  that  TI  items  under  preparation  correctly  reflect 
currently  approved  configurations  of  the  appropriate  hardware 
and  have  taken  into  account  any  engineering  changes;  and 

(4)  evaluate  the  product  on  a  sampling  basis  as  considered 
desirable. 

In-Process  Review  (IPR) 

As  the  term  is  generally  used,  an  IPR  is  a  formal  review  conducted 
by  the  Government  to  examine  TMs ,  TM  source  data,  and  TM  products. 
As  such,  IPRs  are  essentially  an  evaluation  of  the  product  at 
intermediate  stages  of  development.  In  this  Specification,  the 
term  In-Process  Audit  is  used  to  imply  an  increased  function, 
specifically,  a  concern  with  the  Contractor's  adherence  to  his  QAPP 
as  well  as  with  the  quality  of  the  TI  being  developed. 

Interactive  Electronic  Technical  Manual  (IETM) 

A  Technical  Manual,  prepared  (authored)  by  a  Contractor  and 
delivered  to  the  Government,  or  prepared  by  a  Government  activity, 
in  digital  firm  on  a  suitable  medium,  by  means  of  an  automated 
authoring  system;  designed  for  electronic-screen  display  to  an  end 
user,  and  possessing  the  following  three  characteristics: 

(1) The  format  and  style  of  the  presented  information  are  optimized 
for  screen  presentation  to  assure  maximum  comprehension;  that 
is,  the  presentation  format  is  "frame-oriented" ,  not  "page- 
oriented”  . 

(2)  The  elements  of  Technical  Information  constituting  the  TM  are 
so  interrelated  that  a  user's  access  to  the  information  he 
requires  is  facilitated  to  the  greatest  extent  possible,  and  is 
achievable  by  a  variety  of  paths; 

(3)  The  computer-controlled  TM-Display  device  can  function 
interactively  (as  a  result  of  user  requests  and  information 
input)  in  providing  procedural  guidance,  navigational 
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directions,  and  supplemental  information;  and  also  in  providing 
assistance  in  carrying  out  logistic-support  functions 
supplemental  to  maintenance. 

Non-Procedural  TI 

TI  including,  but  not  limited  to,  Descriptive  Information  including 
principles  of  operation,  system  and  component  descriptions;  parts 
lists,  and  other  IPB  information;  source  codes,  functional 
diagrams,  schematics;  and  wiring  data. 

Procedural  TI 

TI  detailing  operation,  maintenance,  and  troubleshooting 
procedures;  including,  but  not  limited  to,  checkout,  calibration, 
alignment,  test,  scheduled  maintenance,  removal,  replacement,  and 
repair . 

Quality  Assurance 

A  planned  and  systematic  series  of  procedures  carried  out  by  the 
Contractor  as  monitored  by  the  Government,  necessary  to  ensure  a 
high  degree  of  confidence  that  deliverable  TI  conforms  to  contract 
requirements.  The  culminating  action  in  a  QA  program  is  Contractor 
support  of  the  Government  Verification  procedure. 

Target  Audience 

Personnel  who  in  terms  of  training,  capability,  and  experience 
closely  resemble  the  Navy  personnel  who  will  actually  use  the  TI . 

Technical  Information  (TI) 

Used  in  this  Specification  to  denote  any  element,  or  combination  of 
elements,  of  information  composing  an  IETM,  IETM  Data  Base,  or  View 
Package.  Thus,  an  IETM  contains,  and  is  composed  of,  Technical 
Information.  In  generating  an  IETM,  an  Author  creates  TI  and  a  QA 
organization  evaluates  the  quality  of  the  TI  as  well  as  of  the 
completed  IETM. 

Technical  Manual  (TM1 

An  organized  collection  of  information,  in  any  form,  and  on  any 
medium,  designed  to  provide  all  guidance  required  by  a  technician 
or  other  end  user  to  carry  out  a  specific  weapon-system  logistic- 
support  function  such  as  maintenance,  operation,  or  training. 

Troubleshooting  TI 

TI  designed  to  permit  fault  isolation  in  corrective  maintenance. 
Troubleshooting  or  Fault-Isolation  TI  is  a  special  type  of 
Procedural  TI  (differing  primarily  in  the  interactivity  involved 
between  Display  Device  and  technician,  and  in  the  presence  of 
extensive  branching  in  the  logic). 

Validation  of  Procedural  TI 

Test  of  TI  against  the  associated  hardware  by  actual  performance  of 
the  procedures  by  target-audience  personnel  using  the  actual 
hardware  or  other  Government- approved  support  equipment  in 
accordance  with  the  Validation  Plan. 


MIL-H-IETMQA 


24 


1  Jum  1990 


MIL-M- IETMQA 
1  June  1990 


IETM  View  Package  Definition 

A  View  Package  is  a  fully  organized  and  formatted  item  of  computer- 
processible  Technical  Information  derived  from  an  IETM  Data  Base 
and  capable  of  interactive  electronic  display  to  an  end  user  by 
means  of  an  Electronic  Display  System  (EDS).  In  function  and 
design,  a  View  Package  is  completely  equivalent  to  an  individual 
Interactive  Electronic  Technical  Manual.  A  View  Package  may  be 
constructed  (1)  entirely  by  an  Author  using  an  automated  authoring 
system;  (2)  completely  automatically  using  a  series  of  automated 
processes  (software)  which  perform  the  data-selection,  structuring, 
and  formatting  processes;  or  (3)  by  a  combination  of  the  two 
approaches.  A  View  Package  is  designed  to  support  a  specific 
function  in  the  operation  or  logistics-support  of  a  weapon  system 
or  other  military  equipment. 


WYSIWYG 

What  You  See  Is  What  You  Get.  Used  in  this  Specification  to 
describe  an  authoring  system  with  the  ability  to  provide  the  Author 
with  a  view  of  an  item  (e.g.,  screen)  of  Technical  Information 
identical  to  the  view  which  the  using  technician  will  see. 
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APPENDIX  A:  SUMMARY  OF  IETM  ACCEPTANCE  TESTS 

This  Section  summarizes  (for  information  purposes  only)  IETM  acceptance 
tests  which  will  be  performed  by  the  Government  on  the  digitized  data 
stream  delivered  by  the  Contractor  for  Government  acceptance  after  all 
deficiencies  observed  during  the  Verification  process  have  been  corrected 
and  re-Validated  as  required.  The  physical  form  of  the  digital  data  stream 
on  delivery  by  the  Contractor  will  be  in  accordance  with  MIL-STD-1840 . 

This  summary  is  provided  for  the  convenience  of  the  IETM  Contractor  so 
that  he  can  assure  that  IETMs  delivered  will  respond  satisfactorily  to  the 
Government's  tests  which  are  intended  to  assure  conformity  of  the  IETM  to 
requirements  of  the  Contract  Specification. 

Provision  of  this  summary  of  acceptance  tests  in  no  way  limits  the 
Government  in  carrying  out  any  additional  tests  that  it  may  deem  advisable 
in  assuring  that  the  IETM  meets  contractual  requirements  and  is  error-free, 
comprehensible,  and  operationally  suitable  for  the  purpose  intended.  Nor 
does  the  Contractor's  own  ability  to  show  a  satisfactory  result  when 
performing  these  tests  show  conclusively  that  the  IETM,  IETM  Data  Base,  or 
View  Package  submitted  is  in  accordance  with  Contract  Specifications. 


A. 1  Test  Systems 

Government  acceptance  tests  will  be  performed  entirely  on  an  Electronic 
Display  System  such  as  that  defined  in  MIL-HDBK-EDS ,  Electronic  Display 
System  for  IETMs,  and  which  meets  all  the  requirements  of  that 
Specification.  Such  a  device  will  have  been  specified  by  the  Government 
and  acquired  and  used  by  the  Contractor  in  Validating  the  IETM  and  in 
performing  other  QA  inspections.  Compatibility  with  an  Electronic  Display 
System  of  this  type  is  a  primary  contract  requirement  for  the  IETM,  since 
the  entire  interactive-display  concept  of  digitized  Technical  Manual 
information  requires  establishment  of  information-calling  sequences  and  the 
exercise  of  special  functions,  with  both  processes  dependent  on  software 
which  is  functionally  common  to  the  automated  authoring  system  and  to  the 
Display  Device.  The  proof  that  an  IETM  is  satisfactory,  therefore,  is  that 
it  may  be  scrutinized  in  a  way  which  demonstrates  conformance  with  all 
Specification  requirements  on  the  Device  which  the  end-user  (e.g.,  a  Fleet 
or  flight-line  technician)  will  use  to  view  it. 


A. 2  Test  Sequence 

Tests  will  be  conducted  in  two  . phases: 

a.  Tests  to  assure  that  the  IETM  functions  satisfactorily  when  viewed 
with  a  test  Display  Device;  specifically,  that  all  special 
functions  are  available,  calling  sequences  are  correct,  and  in 
general  that  all  capabilities  requested  have  in  fact  been  provided. 

b.  A  thorough  review  of  the  content  of  each  screen  to  assure  that  no 
procedures  have  been  left  out,  that  the  material  is  written 
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comprehensibly,  that  the  Technical  Information  is  error-free,  that 
troubleshooting  Information  in  fact  leads  to  Fault  Isolation  in  all 
branches,  and  that  the  technician  is  informed  as  to  the  calling 
sequences  for  required  information  and  can  obtain  it  quickly  and 
directly. 

These  tests  are  outlined  in  greater  detail  below. 


A.  3  Tests  of  IETM  Functions 


A. 3.1  Screen  Sequencing 

Using  the  required  calling  algorithm  based  on  screen  references,  every 
screen  in  the  IETM  will  be  viewed  from  the  first  to  the  last,  and  the 
logical  sequences  applicable  to  the  Maintenance  Procedure  and  to 
Troubleshooting  will  be  evaluated. 


A. 3. 2  Special  Functions 

Each  of  the  special  functions  provided  in  the  Electronic  Display  System 
(specified  in  Section  3.4  of  MIL-M-GCSFUI )  will  be  checked  to  see  that  it 
performs  properly  with  the  IETM  being  evaluated. 

The  ability  of  a  technician  to  exercise  these  functions  using  the 
function  keys  and  touch-screen  capability  of  the  Display  Device  will  be 
evaluated. 


A. 3. 3  Display  Characteristics 

Tests  outlined  in  this  Section  are  not  tests  of  the  Display  System  but 
of  the  IETM.  The  individual  screens  shall  be  displayed  so  that  they  can 
be  seen  and  understood  properly  by  the  technician,  within  the  limitations 
of  the  screen  size,  resolution,  etc.  The  visibility  of  each  screen 
provided  will  accordingly  be  checked  in  this  context.  (This  test  was  also 
performed  during  Verification.) 


A. 3. 4  Capability  to  Access  Information 

One  of  the  primary  reasons  for  preparation  of  IETMs  is  to  provide  a 
technician  with  the  ability  to  reach  needed  information  immediately, 
without  searching  through  hundreds  of  paper  pages.  For  this  reason,  the 
IETM  is  designed  to  provide  multi-path  calling  methods  to  access  the 
information  required  directly  (i.e.,  without  proceeding  via  numerous 
successive  menus).  The  ability  of  the  technician  to  retrieve  the 
information  he  needs  for  specific  problems  will  accordingly  be  evaluated 
based  on  a  number  of  input  requests,  as  described  in  MIL-M-GCSFUI. 
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The  Government  will  develop  a  software  routine  capable  of  being  used  to 
test  exhaustively  this  aspect  of  the  Automated  TI  when  used  with  the 
Government-Specified  or  Government-Furnished  Electronic  Display  Device,  to 
assure  (1)  that  all  calling  sequences  (data-access  links)  established  by 
the  Author  are  valid  and  operable;  (2)  that  calling  sequences  implied  by 
special  function  keys  are  fully  operable;  (3)  that  user-established 
calling  sequences  perform  as  indicated;  and  (4)  that  there  are  no  "lost" 
(unreachable)  data  in  the  entire  body  of  TI.  This  software,  which  will  be 
identical  (or  very  nearly  so)  to  a  software-test  routine  which  will  be  a 
permanent  part  of  the  fielded  Electronic  Display  Device,  will  be  made 
available  to  the  Contractor  to  aid  him  in  continued  performance  of  QA 
checks.  (See  MID-D-EDS,  Section  3.9.) 


A. 4  Evaluation  of  IETM  Content,  Format,  and  Style 

Selected  screens  will  be  used  to  perform  a  detailed  evaluation  of  TM 
Content,  Format,  Style,  and  User-Interaction  characteristics .  For  example, 
a  number  of  Maintenance  Tasks  will  be  followed  through  each  Subtask  and 
Step  to  evaluate  completeness,  accuracy,  competence  of  the  graphics, 
arrangement,  conformance  to  Style  requirements,  inclusion  of  Warnings, 
Cautions,  and  Notes,  completeness  of  the  setup  information,  and  the  like. 


A. 4.1  Evaluation  of  Troubleshooting  TI 

Beginning  with  operationally  realistic  defective-performance  reports,  a 
series  of  troubleshooting  paths  (including  Fault  Verification)  will  be 
followed  in  detail  to  establish  the  effectiveness  of  these  test  sequences 
in  isolating  the  fault.  Experienced  technicians  will  evaluate  not  only  the 
mechanical  effectiveness,  but  also  the  technical  competence  of  the 
troubleshooting  procedure  designed  by  the  Author  for  Fleet-system 
Troubleshooting  under  operational  conditions . 
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